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Evaluation Goals and Objectives 


Over the past decade a significant amount of research on effective reading instructional 
practices for beginning and struggling readers has been conducted. The authors of a 1998 
report published by the National Research Council concluded that phonics instruction that is 
systematic and explicit is an effective means of teaching children to read at the word level.! In 
2000 the National Reading Panel conducted a meta-analysis of quantitative studies of phonics 
instruction and determined that systematic phonics instruction helped students learn to read 
better than all forms of control group instruction. The panel concluded that systematic and 
explicit phonics instruction should be incorporated into literacy programs for beginning readers 
and into programs for preventing and remediating reading difficulties in struggling readers. 
Since 2000 numerous instructional programs that incorporate scientific reading research have 
been developed. In general, the effectiveness of these programs has been difficult to ascertain. 
Many have not undergone evaluations, and many of the evaluations that have been conducted 
employed research designs that lacked rigor. 


Intervention Model 


Scholastic’s System 44 is a recently developed foundational reading program intended for older 
struggling readers who have not mastered basic phonics and decoding skills. Combining 
researched-based phonics instruction with adaptive technology, System 44 is designed to 
improve students’ word reading accuracy, fluency, and comprehension. The System 44 program 
delivers research-based instruction through an adaptive computer component; teacher-led small 
group instruction; and individual student practice involving high-interest, leveled materials. Thus 
students who have not responded to classroom reading instruction may benefit from the more 
intensive and specific decoding instruction provided through System 44. 


Students in the System 44 classroom are expected to receive approximately 20 to 25 minutes of 
computer-delivered instruction, complete 25 to 30 minutes of small group and individual work, 
and receive 5 to 10 minutes of whole class instruction each day. The program includes 25 
series, each of which covers 5 to 8 topics, each of which comprises 4 strands: The Code 
(decoding instruction), Sight Words (high-frequency words), Word Strategies (syllable types and 
word parts), and Success (reading connected text). Beginning with Series 4, students complete 
a preliminary Fast Track Assessment to determine whether they have already mastered the 
skills covered in the series and can skip to the next series. Each software lesson in a series has 
a set of corresponding practice activities in (a) the 44Book, (b) the Decodable Digest, and 

(c) the 36-title System 44 paperback book and audiobook library. Supplementary instructional 
materials include practice worksheets from the online Scholastic Achievement Manager (SAM), 
letter tiles, posters, flip chart lessons, and the DVD Sound and Articulation. Students track their 
progress on these materials using the Self-Monitoring Chart. Approximately twice a week the 
teacher leads a 5- to 10-minute whole class instructional activity. The teacher can use SAM to 
generate reports to track individual student progress on each skill and identify students in need 
of small group instruction. 


"Snow, I.C.E., Burns, S., & Griffin, P. (Eds.). (1988). Preventing reading difficulties in young children. Washington, 
DC: National Research Council, Committee on the Prevention of Reading Difficulties in Young Children. 

?National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel. Teaching 
children to read: An evidence-based assessment of the scientific research literature on reading and its implications 
for reading instruction (NIH Publication No. 00-4769). Washington, DC: U.S. Government Printing Office. 
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Logic Model 


Exhibit 1 portrays a logic model that summarizes the System 44 program inputs, classroom 
practices, and expected outcomes for participating teachers and students. 


Overview of the Evaluation 


Using a randomized design, this evaluation assessed the effectiveness of System 44 in terms of 
improving the foundational reading skills of struggling readers in Grades 4—8 in a large 
suburban school district that served approximately 22,000 students in 18 schools: 11 
elementary schools, 4 middle schools, and 3 high schools. The evaluation of the implementation 
and impact of System 44, which involved 7 of the elementary schools and all 4 middle schools 
in the district, sought to answer 5 research questions: 


1; 


What contextual factors are involved in the implementation of System 44 (i.e., factors 
that promote or hinder successful implementation of the program)? 


What are teachers’ perceptions of the program? 


What are the effects of System 44 on student outcomes? Specifically, how do changes 
in word reading accuracy, fluency, and comprehension achieved by System 44 students 
compare to changes achieved by a control group? 


How does System 44 differentially affect subgroups of students? Specifically, how do 
changes in word reading accuracy, fluency, and comprehension achieved by specific 
subgroups of System 44 students (English language learners, students in special 
education, students receiving free or reduced-price meals, and ethnic minority students) 
compare to changes achieved by subgroups of control group students? 


What is the association between System 44 effects and program implementation? 
Specifically, are changes in System 44 participants’ word reading accuracy, fluency, and 
comprehension associated with variation in program implementation (including time on 
software, time in small groups, and resources available to teachers and students) ? 


Evaluation of System 44 


Program 
Inputs/Activities 


Professional development 
for teachers (beginning of 
year). 


In-school coaching by 
Scholastic (3 sessions per 
teacher). 


District meetings for 
System 44 teachers to 
share experiences. 


Curriculum materials: 


System 44 software. 
Decodable Digest. 
44Book. 


System 44 paperback 
book and audiobook 
library. 


Other (practice 
worksheets, letter 
tiles, posters, flip 
chart, Sound and 
Articulation DVD). 


Logic Model for System 44 


Classroom 
Practices and 
Teacher Activities 


Daily use of 
Scholastic System 44 
software with fidelity. 


Whole class 
instruction for 

5 minutes at 
beginning of class. 


Small group 
instruction. 


Differentiated 
instruction. 


Opportunities for 
students to practice 
skills independently. 
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Teacher 
Outcomes 


Fidelity of System 44 
implementation: 


Classroom setup. 


Minutes of 
instruction. 


Use of program 
components. 


Student grouping. 


Satisfaction with 
professional 
development and 
support. 


Perceived 
effectiveness of 
program. 


Short-Term 
Student 
Outcomes 


Total System 44 
topics completed. 


Improved decoding 
accuracy, sight word 
accuracy, and 
comprehension 
scores. 


Long-Term 
Student 
Outcomes 


Improved reading 
fluency and 
comprehension. 


Improved 
performance on state 
reading test. 
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Evaluation Methods 


Scholastic implemented System 44 in 11 elementary and middle schools in a large suburban 
school district in southern California during the 2010—2011 school year. The evaluation used a 
randomized trial design whereby RMC Research randomly assigned eligible students to a 
treatment (System 44) group or a control (“services as usual”) group. 


Sampling Plan 


A 2-step process was used to establish student eligibility for System 44. The Scholastic Reading 
Inventory (SRI) was used to screen students in Grades 4—8 who performed below the 50th 
percentile on the spring 2010 California Standards Test (CST) for System 44 eligibility. Those 
students who scored below 600 Lexiles® on the SRI were administered the Scholastic Phonics 
Inventory (SPI), a computer-based test used to identify students in need of additional phonics 
instruction. Students who scored in the Beginning or Developing reader categories on the SPI 
were randomly assigned (stratified by school and grade level) to either the System 44 treatment 
group or the control group. 


Participants 


A total of 425 students met the eligibility criteria for System 44 and were randomly assigned to 
the treatment group (n = 216) or the control group (n = 209). Exhibit 2 summarizes the 
characteristics of all treatment and control group students in the randomized sample and those 
students who were included in the analytic sample. The evaluation team conducted equivalence 
tests on key factors to determine whether differences between the treatment and control groups 
existed at baseline. Overall, the treatment and control groups in both the randomized and 
analytic samples were equivalent in terms of sex, eligibility for free or reduced-price meals, 
English language proficiency, special education status, ethnicity, and baseline CST scores. 


5For more information about the Lexile Framework for Reading see http://www.lexile.com/ 
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Exhibit 2 
Student Characteristics 


Randomized Sample Analytic Sample 
Treatment Control Treatment Control 
Characteristic (n = 216) (n = 209) (n = 173) (n =171) 
Grade Level 
4 1 (42%) 6 (41%) 73 (42%) 2 (42%) 
5 0 (14%) 1 (15%) 5 (14%) 7 (16%) 
6 5 (26%) 3 (25%) 43 (25%) 9 (23%) 
7 2 (10%) 4 (12%) 0 (12%) 1 (12%) 
8 8 (8%) 5 (7%) 2 (7%) 2 (7%) 
Sex 
Male 120 (60%) 116 (58%) 106 (61%) 95 (56%) 
Female 80 (40%) 84 (42%) 67 (39%) 75 (44%) 
Free or Reduced-Price Meals* 
None 109 (54%) 119 (60%) 97 (56%) 103 (61%) 
Reduced price 67 (34%) 66 (33%) 58 (34%) 56 (33%) 
Free 24 (12%) 15 (7%) 18 (10%) 11 (6%) 
English Learner Status* 
English only 132 (66%) 143 (72%) 114 (66%) 124 (73%) 
Initial full English speaker 2 (11%) 6 (8%) 1 (12%) 4 (8%) 
English learner 2 (16%) 9 (14%) 5 (14%) 0 (12%) 
Redesignated former English learner 4 (7%) 2 (6%) 3 (8%) 2 (7%) 
student 
Special Education Status 
None 135 (62%) 136 (65%) 120 (69%) 121 (71%) 
Specific learning disability 6 (7%) 7 (8%) 2 (7%) 3 (8%) 
Speech or language impairment 2 (10%) 7 (138%) 7 (10%) 1 (12%) 
Other classification 43 (20%) 9 (14%) 4 (14%) 6 (9%) 
Primary Ethnicity®° 
Caucasian 123 (62%) 129 (64%) 107 (62%) 109 (64%) 
African American 9 (10%) 9 (10%) 5 (9%) 8 (11%) 
Hispanic 3 (17%) 4 (17%) 9 (17%) 6 (15%) 
Asian/Pacific Islander 2 (6%) 0 (5%) 0 (6%) 9 (5%) 
Filipino 2 (6%) 7 (4%) 1 (6%) 7 (4%) 
Other 0 (0%) 1 (0%) 0 (0%) 1 (1%) 


*Data missing for 25 students | in the randomized sample (16 treatment, 9 control) and 1 student in the analytic 
sample control group. One treatment group student declined to respond. 
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Of the 425 randomly assigned students, 345 (172 treatment, 173 control) received the allocated 
intervention or control group condition as planned. Of the 44 treatment group students and 36 
control group students who did not receive the allocated intervention or control group conditions 
as planned, 14 (8 treatment, 6 control) were ineligible for participation in the study due to 
exposure to System 44 the previous year, a fact that was unknown at the time of random 
assignment. An additional 25 students (10 treatment, 15 control) were mistakenly assigned— 
that is, the schools already had assigned those students to Scholastic’s Read 180 program and 
had not intended them to be included in the eligibility pool. Other reasons students did not 
receive the allocated intervention or control group condition included parent withdrawal (1 
treatment, 3 control), irreconcilable scheduling issues (4 treatment), skills deemed too high for 
the intervention after random assignment occurred (4 treatment), transfer to a non-System 44 
school (17 treatment, 7 control), and crossover (5 control students received the System 44 
intervention). Additionally, 1 control student who received the allocated intervention refused to 
participate in the study testing. A total of 344 of the 425 randomly assigned students (81%) were 
retained in the final analytic sample for all individual test outcomes. One school administered 
posttest SRI tests to a subset of treatment and control group students and administered posttest 
SPI tests to a subset of treatment students. The treatment and control group students who 
completed the SRI posttest were retained in the SRI outcome analyses, but the school’s data 
were excluded from all SPI outcome analyses. Thus the final SRI analytic sample was 332 and 
the final SPI analytic sample was 293. 


To assess whether selective study attrition occurred in the analytic sample, the evaluation team 
conducted equivalence tests on baseline demographic characteristics and CST reading scores. 
No differences were evident at baseline in terms of sex, eligibility for free or reduced-price 
meals, and ethnicity between students who were retained in the analytic sample and students 
who were not. However, the students retained in the analytic sample did have higher baseline 
CST scores than those students who were not retained. Students retained in the final analytic 
sample were also less likely than students who were not retained to have a special education 
classification other than learning disability or speech or language impairment and were more 
likely to be former English language learners and less likely to be current English language 
learners. 


Data Collection 


Data collection activities for the System 44 evaluation included student reading tests, teacher 
surveys, System 44 classroom observations, a professional development observation, and staff 
interviews. Survey, observation, and interview materials are included in Appendix A. 


Student Reading Tests 


RMC Research hired and trained 4 local testers to administer a battery of standardized reading 
tests to all treatment and control students . The testers administered the tests to each student 
separately over a 3-week period in September and October 2010 to establish baseline scores 
and again in May 2011 to attain follow-up scores. Listed in order of administration, the tests 
included the following: 


« Test of Silent Reading Efficiency and Comprehension (TOSREC)—A 60-item 
comprehension test that assesses a student’s ability to silently read a sentence and 
identify whether the sentence is true or false. The test is grade level specific. Form A 
was administered in fall 2010, and Form C was administered in spring 2011. 
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= Comprehensive Test of Phonological Processing (CTOPP) Elision subtest—A 
34-item test that assesses the extent to which a student can say a word and then say 
what is left of the word after being instructed to drop out designated sounds. 


" Woodcock-Johnson Ill Word Identification subtest—An untimed test that assesses a 
student’s ability to accurately read aloud printed words. 


" Woodcock-Johnson Ill Word Attack subtest—An untimed test that assesses a 
student's ability to apply phonic and structural analysis skills by measuring the number of 
non-words that a student can accurately pronounce aloud. 


« Test of Word Reading Efficiency (TOWRE) Sight Word Efficiency subtest—A 
108-item test that measures the number of printed words that a student can accurately 
read aloud in 45 seconds. Form A and Form B were administered back-to-back in fall 
2010 and spring 2011. 


" Test of Word Reading Efficiency (TOWRE) Phonetic Decoding Efficiency subtest— 
A 66-item test that assesses a student’s ability to apply grapho-phonemic knowledge by 
reading pronounceable non-words. The test measures the number of non-words that a 
student can accurately read aloud in 45 seconds. Form A and Form B were administered 
back-to-back in fall 2010 and spring 2011. 


The battery of standardized reading tests required approximately 35 to 45 minutes to administer 
to each student, which included time to build rapport. The test administrators reported that 
although the testing went smoothly overall, a few students were difficult to test due to language 
barriers or other reasons such as visual impairment or refusal to participate. Other tests 
administered to the participating students included the English-Language Arts subtest of the 
CST; the SRI, a test of reading comprehension skills; and the SPI, a test of accuracy and 
fluency in letter recognition, sight word recognition, and non-word decoding. 


Teacher Survey 


A survey administered to System 44 teachers in fall 2010 and spring 2011 gathered information 
about the teachers’ background, the types of instructional strategies and activities they 
employed to teach foundational reading skills, their opinions about the effectiveness of 

System 44, their opinions about System 44 professional development, and their knowledge of 
the System 44 program. For comparative purposes, a sample of control group teachers 
completed a modified version of the survey in spring 2011. 


Classroom Observations 


RMC Research staff observed every System 44 classroom 3 times during the 2010-2011 
school year (November, February, and April) to gather data on System 44 implementation. The 
classroom visit protocol captured the essential features of System 44 and other features of 
effective instruction such as maintaining a positive learning environment, monitoring students, 
and providing differentiated support based on student needs. 


Professional Development Observations 


RMC Research staff also observed the professional development provided to System 44 
teachers at the beginning of the school year to better understand the expected implementation 
of the program and any training issues that might have interfered with the fidelity of program 
implementation. 
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Staff Interviews 


The end-of-year interviews with teachers, principals, and district staff assessed the contextual 
environment in which System 44 was implemented, the factors that facilitated or hindered 
implementation, the fidelity of System 44 implementation, and staff perceptions of System 44 
materials and instructional activities. 


Analysis Plan 


RMC Research utilized a mixed-method approach to evaluate the impact of System 44. 
Program implementation was evaluated through quantitative analyses of classroom 
observations and teacher surveys; these analyses were supported by qualitative information 
gathered through interviews with district staff, school principals, and System 44 teachers. To 
measure the impact of System 44 on student test scores, RMC Research used a series of 
quantitative analyses. 


Implementation Analysis 


Classroom implementation fidelity was defined as the average of the fidelity ratings that were 
part of the classroom observations conducted by the evaluation team in fall 2010, winter 2011, 
and spring 2011. 


Implementation fidelity included the following components: 


« Classroom setup. 

« Minutes of instruction on software and in small groups. 
» Inclusion of all program components. 

« Instructional management and delivery. 


Teacher survey data were used to examine: 


«Teacher background characteristics. 
= Teacher perceptions of System 44 professional development. 


« Reading instructional practices prior to System 44 compared to reading instructional 
practices using System 44. 


« Reading instructional practices of System 44 teachers compared to reading instructional 
practices of control group teachers. 


« Perceptions of the effectiveness of reading instructional practices prior to and using 
System 44. 


« Frequency of use of System 44 components. 
« Understanding of System 44 program concepts and components. 


Teacher interview data from spring 2011 provided detailed information concerning the teachers’ 
perceptions of the strengths and challenges of using System 44, how they made decisions 
concerning student grouping, and how they used various SAM reports. Principal interview data 
from spring 2011 provided information about the strengths and challenges of implementing 
System 44 and other contextual issues. District staff interview data from spring 2011 provided 
additional information about the professional development, the factors that hindered or 
facilitated implementation of System 44, perceptions of program effectiveness, and 
recommendations for future implementation. 
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Impact Analysis 


This evaluation used an intent-to-treat statistical model—a framework in which participants are 
analyzed within their initial random assignment group regardless of whether they actually 
received treatment. Because students were clustered within schools, a multilevel model was 
used to estimate the impact of the intervention on spring 2011 outcome scores while controlling 
for baseline score, ethnicity, English language proficiency, and school level. The model was run 
separately for each outcome measure. The school district provided individual student 
demographic data for grade level, sex, free or reduced-price meal eligibility, ethnicity, special 
education status, and English language proficiency. The following demographic covariates were 
included in the model: ethnicity (Caucasian), special education status, and sex. In addition, 
school level was added as a covariate. No data were missing for any of the demographic 
covariates. To assess the impact of System 44 on student outcomes, the evaluation team used 
hierarchical linear modeling, controlling for baseline scores and student demographic 
characteristics at Level 1 and school at Level 2. The 2-level model for estimating the impact of 
System 44 on change in outcomes is specified below. 


Level 1 Model: 
Yi = Boj + B4)(Trtj) + B2(Pretest;) + B3(SPED;j) + B4(Caucasian;) + B5(Sexi) + Bg(Leveli) + &j 


Level 2 Model: 
Boj = Yoo + Hoj 
Bi = Yo 
Bo = Y20 
Bs = Y30 
Bsa = Yao 
Bs = Y50 
Be = Yeo 
where: 
gj = the random error term for student /in school / 
lo) = the random intercept term for school / 
and: 
Yi = the posttest score for student /in school / 
Trtj = the treatment indicator for student /in school / 
(0 = control, 1 = treatment) 
Pretest; = the posttest score for student /in school / 
SPED, = the special education indicator for student /in school / 
(0 = no special education classification, 1 = special education 
classification) 
Caucasian; = the ethnicity indicator for student /in school / 
(0 = non-Caucasian, 1 = Caucasian) 
Sexi = the sex indicator for student /in school / 


(0 = male, 1 = female) 
Leveli = the school level indicator for student /in school / 
(0 = middle, 1 = elementary) 
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Program Implementation Findings 


To provide context for interpreting the impact of the System 44 intervention, the evaluation team 
assessed its implementation from several perspectives. Scholastic staff expected 
implementation to improve as teachers gained experience using System 44. Although many 
factors contributed to the fidelity of implementation, barriers to implementation also emerged. 
This section summarizes the findings for the 2 evaluation questions pertaining to program 
implementation. 


Contextual Factors in System 44 Implementation 


Data regarding contextual factors involved in the implementation of System 44 were obtained 
from the teacher surveys; the interviews with teachers, principals, and district staff; and the 
observations of System 44 classrooms. These data include descriptions of the System 44 
classrooms and the counterfactual, teacher background characteristics, the professional 
development activities, and the fidelity of System 44 implementation. 


System 44 Classrooms 


Scholastic staff expected teachers using the System 44 model in a 60-minute class period to 
allocate their time such that students spent approximately 20 minutes on the System 44 
software; approximately 20 minutes reading System 44 library books, completing book logs, and 
taking Scholastic Reading Counts quizzes; and approximately 20 minutes in whole class or 
small group instruction involving SMART lessons, the 44Book, or the Decodable Digest or 
conferencing individually with the teacher. One middle school (2 classrooms) had a shorter 
48-minute period that allocated 20 minutes for System 44 software and the remaining 28 
minutes for System 44 library books and small or large group instruction. 


The Counterfactual 


The counterfactual varied across schools because students in the control group did not receive 
a uniform alternative to the System 44 intervention. The difference was greatest between the 
elementary schools, which used a pull-out model, and the middle schools, which used a 
replacement model. Because System 44 was a pull-out class at the elementary school level, the 
control group students who remained in the regular classroom engaged in a variety of activities 
across content areas. In contrast, in the middle schools System 44 was delivered during the 
language arts instruction block and a key difference between the treatment and control groups 
was lower student-teacher ratios in the System 44 classes. Although some System 44 teachers 
also taught control group students, they did not use System 44 materials with the control group. 


Appendix B summarizes the reading interventions provided in the control group classrooms 
during the 2010-2011 school year. The elementary school reading interventions included 
Voyager-Passport, Ticket to Read, Read Naturally, SRA Decoding, and Houghton-Mifflin. The 
middle school reading interventions included Voyager-Journeys, Houghton-Mifflin, Rosetta 
Stone, Explode the Code, and various worksheets. 
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Teacher Background Characteristics 


The teacher survey administered to both the treatment group and the control group solicited 
information about the respondents’ teaching experience, education, certifications, and prior 
experience with Scholastic’s System 44 and Read 180 programs. All but one of the System 44 
teachers (92%) were female, and 73% of the control group teachers were female. Exhibit 3 
shows that the control group teachers had more teaching experience than the System 44 
teachers and were more likely to have a degree beyond a Bachelor’s degree. The System 44 
teachers were, however, more likely to have Education Specialist or Reading Specialist 
certification and to have prior experience teaching System 44 or Read 180. 


Exhibit 3 
Teacher Education and Experience 
Treatment Control 
Characteristic M SD Percent M SD Percent 
Teaching Experience 
Total years of teaching experience 12.2 6.3 14.3 8.6 
Years teaching in this district 7.8 5.1 10.6 6.1 
Years teaching at current school 5.8 3.0 8.3 6.2 
Education (Highest Degree) 
Bachelor’s degree 33% 24% 
Master’s degree 42% 61% 
Specialist or doctorate 25% 15% 
Certifications 
Professional Clear 67% 55% 
Education Specialist 33% 18% 
Reading Specialist 8% 3% 
Administrative Services 17% 6% 
CLAD or BCLAD? 75% 64% 
Prior Scholastic Experience 
System 44 50% 12% 
Read 180 83% 18% 


Note. n= 12 System 44 teachers; n = 33 control group teachers 
*CLAD = Crosscultural, Language, and Academic Development. BCLAD = Bilingual, Crosscultural, Language, and 
Academic Development. 


Professional Development 


Professional development for the System 44 teachers included a 1-day training in August 2010, 
3 coaching visits from Scholastic staff during the 2010-2011 school year, and periodic district 
meetings that served as a System 44 professional learning community for participating 
teachers. Analyses of the professional development are based on the training observation 
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results, information obtained from the Scholastic trainer at the conclusion of the training, and the 
teacher survey results. 


In August 2010 Scholastic staff and school district staff provided a 1-day training for 24 teachers 
who were likely to use the System 44 program. Most of the teachers reported basic familiarity 
with System 44, and the trainer provided only a brief overview of the program. The overview 
sufficiently addressed the intervention components and the student learning goals and 
objectives and lightly covered the theoretical and empirical support for the intervention content. 
The trainer addressed questions about specific implementation requirements for the evaluation 
(e.g., the System 44 class period needed to be 60 minutes long and students needed at least 20 
minutes on the computer daily). The trainer introduced the teachers to the breadth of the 
instructional materials but did not have sufficient time to cover the materials in depth. Most of 
the teachers wanted to focus on the software and most seemed to gain a good understanding of 
how students would progress through the program. The trainer addressed strategies for 
integrating the intervention into the classroom, utilizing the embedded tests and highlighting key 
instructional skills, but the teachers spent little time practicing the skills. Overall, the trainer 
provided a quality, interactive training that was tailored to the teachers’ needs, covered the 
breadth of the program materials, and addressed the teachers’ questions and concerns. A 
longer training would have allowed the trainer to cover the materials in greater depth and offer 
more time for teachers to practice key instructional strategies. 


At the conclusion of the training the teachers responded to several survey questions. Half 
reportedly had not received any previous training on System 44, 25% had attended similar 
training in a previous school year, and 25% had received other support for teaching System 44 
from a Scholastic representative in a prior year. System 44 teachers again responded to survey 
questions about professional development activities in spring 2011. Exhibit 4 summarizes the 
System 44 teachers’ responses at both time points. Overall, the teachers gave the professional 
development high ratings—although the mean ratings were somewhat lower in spring 2011 than 
in fall 2010. Several teachers noted that the training on SAM reports was particularly helpful. 


Exhibit 4 
Ratings of System 44 Professional Development 
Fall 2010 Spring 2011 
Professional Development Characteristic M SD M SD 
The teacher training prepared me to use the 4.75 0.45 4.17 de'Q 
program in my classroom. 
The individual support from Scholastic during the — — 4.09 1.14 
year enhanced my skills in using System 44 in 
my classroom. 
The individual support from district staff during — — 3.91 1.22 


the year enhanced my skills in using System 44 
in my classroom. 


| am pleased with the amount of System 44 4.42 0.67 3.91 1.22 
professional develop | received. 
| am pleased with quality of System 44 4.75 0.45 4.18 1.17 


professional development | received. 


Note. A dash indicates that the question was not asked at that time point. Ratings based on a 5-point scale that 
ranged from 1 (strongly disagree) to 5 (strongly agree). Number of respondents = 12. 
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Fidelity of Implementation: Classroom Observations 


The evaluation team assessed the fidelity of System 44 implementation in 2 ways: classroom 
observations and teacher self-report in surveys and interviews. This section summarizes the fall, 
winter, and spring classroom observations in terms of classroom setup, minutes of instruction, 
inclusion of program components, instructional management, and instructional delivery. 


Classroom Setup. Classroom setup was rated on 3 criteria: computers for at least one third of 
the class are accessible and functioning, auxiliary equipment (headsets, microphones, CD 
players) are accessible and functioning, and System 44 materials are easily accessible to 
students and teachers. The mean rating across these items was converted to a 4-point scale. In 
fall 2010, 58% of the 12 classrooms observed received the highest rating; this percentage 
increased to 75% in the winter and 100% of the classrooms in spring 2011. 


Minutes of Instruction. Minutes of System 44 instruction were consistent across all 3 
observations. Although 9 of the 12 classrooms provided at least 55 minutes of instruction daily, 
3 middle school classrooms provided between 45 and 49 minutes of instruction daily because 
the class period length did not allow for the allocation of 60 minutes for System 44 instruction. 


Inclusion of Program Components. During small group instruction and individual work time 
students were expected to use 3 primary program components on most days: the 44Book, the 
Decodable Digest, and the System 44 paperback book and audiobook library. In addition, on 
some days students used letter tiles and SAM practice worksheets or teachers used the 
System 44 flip chart to instruct small groups. Exhibit 5 summarizes the materials used during 
the observed teacher-led small group instruction, and Exhibit 6 summarizes the materials used 
during the observed independent work time. In most cases half of the class used the computer 
software while the other half received small group instruction or worked independently, and 
midway through the class period the groups changed places. 


During teacher-led small group activities, observers most frequently noted the use of the 
System 44 flip chart, letter tiles, and the Decodable Digest. While students worked 
independently, observers most frequently noted the use of the 44Book, System 44 paperback 
books, and the Decodable Digest. The mean number of students using each type of material 
declined over time due to decreasing class size as students completed the program. Level of 
student engagement appeared to peak at the winter observation point. 
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Exhibit 5 
Use of Instructional Materials During Teacher-Led Small Group Activities 


Total Number of 


Times Observed Mean Level of Use 
Instructional Item Fall Winter Spring Fall Winter Spring 
Decodable Digest 7 1 1 1.50 1.00 2.00 
44Book 4 0 0 1.50 — — 
Teaching resources for 0 0 0 — — — 
System 44 library 
Letter tiles 5 4 2 2.00 2.00 1.50 
System 44 flip chart 7 6 6 1.75 1.88 1.88 
Sound and Articulation DVD 0 0 0 — — — 
SAM resources/worksheets 2 2 2 1.00 2.00 2.00 
System 44 library books 4 0 0 1.25 — — 
SAM reports 0 0 3 — — 1.25 


Note. Classrooms observed: 12. Total number of times observed is out of 25 rotations in fall, 22 in 
winter, and 23 in spring. Rating scale for level of use: 0 = none; 1 = partially; 2 = fully. Dashes indicate 
that the item was not observed. 


Exhibit 6 
Use of Instructional Materials in Independent Work 


Total Number of Mean Number of Mean Level of 

Times Observed Students Using Student Engagement 
Type of Material F W Ss F W Ss F W Ss 
System 44 paperback 13 17 13 6.8 3.9 4.7 1.6 1.8 1.7 
books 
System 44 audiobooks 7 11 7 2.2 2.2 1.4 1.5 1.9 2.0 
44Book 13 17 16 7.0 4.6 3.3 1.4 1.8 1.6 
Decodable Digest 9 15 15 5.0 4.4 4.0 1.2 1.7 1.7 
SAM resources/ 7 13 5 7.0 6.5 2.7 1.6 1.8 1.7 
worksheets 


Note. Classrooms observed: 12. Total number of times observed is out of 25 rotations in fall, 22 in winter, and 23 in 
spring. F = fall, W = winter, S = spring. Rating scale for level of student engagement: 0 = Not at all engaged/none 
engaged; 1 = partially engaged/some engaged; 2 = fully engaged/all engaged. 


Instructional Management and Delivery. Each System 44 class observed was rated on 6 
aspects of instructional management pertaining to the teacher’s ability to maintain a positive 
learning environment, monitor students, keep students on task, provide smooth transitions 
between rotations, use flexible student groups, and differentiate support based on student 
needs. Each class was also rated on 5 aspects of instructional delivery pertaining to pacing, 
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appropriateness of the delivery for the student skill level, teacher preparation, active 
engagement of the students, and overall lesson execution. Exhibit 7 summarizes the mean 
ratings for instructional management and delivery across the 3 observation time points. Overall, 
the observers rated instructional management and instructional delivery lowest in fall 2010 and 
highest in winter 2011. 


Exhibit 7 also provides mean ratings of the 11 items that compose the 2 scales. The aspects of 
instructional delivery that received the lowest ratings were “appear well prepared for the lesson” 
and “execute the lesson well.” The aspects of instructional management that received the 
lowest ratings were “use flexible groupings for students based on instructional needs” and 
“differentiate support based on students’ needs.” The ratings on these instructional 
management aspects were lowest in fall 2010 (Ms = 0.8 and 1.6, respectively) and higher in 
winter 2011 (Ms = 1.6 and 2.4, respectively). 


Exhibit 7 
Ratings of Instructional Management and Delivery 
Fall Winter Spring 

Scale M SD M SD M SD 
Instructional Management 2.8 1.10 3.6 0.67 3.5 0.69 
Instructional Delivery 3.0 1.30 3.3 0.89 3.0 1.50 
Instructional Management 

Maintain positive learning environment 2.7 0.65 28 0.39 2.7 0.47 

Monitor students 2.6 0.79 27 0.49 2.6 0.50 

Keep students on task 2.6 0.79 2.7 0.49 2.6 0.50 

Ensure smooth transitions between rotations 2.2 0.75 26 0.67 2.6 0.50 

Use flexible groups for students based on 0.8 1.20 1.6 1.30 1.7 1.30 

instructional needs 

Differentiate support based on students’ needs 1.6 1.30 24 0.90 2.4 0.93 
Instructional Delivery 

Maintain an appropriate pace 2.6 0.52 26 0.52 2.9 0.33 

Deliver lessons appropriate for skill levels of students 2.7 048 28 0.39 2.9 0.33 

Appear well prepared for lesson 2.1 1.20 24 0.90 1.8 1.40 

Actively tries to engage students 2.0 1.30 2.5 0.90 2.0 1.50 

Executed the lesson well 2.0 1.20 2.2 0.87 1.9 1.40 


Note. Rating scale: 0 = not at all; 1 = to a small extent; 2 = to a moderate extent; 3 = definitely. Number of 
classroom observations = 35. 


Fidelity of Implementation: Teacher Self-Report 


The spring 2011 teacher survey respondents reported that students used the System 44 
software between 20 and 30 minutes daily (average across classrooms: 25.3 minutes). The 
number of computers available in each classroom ranged from 5 to 20 (average 8.8). Exhibit 8 
shows the reported frequency of use for a range of System 44 components. SAM reports, 
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individual student support, and Reading Counts quizzes were used most frequently. Teachers 
used the bound version of the Teaching Guide more frequently than they used the computer 
version that was part of Scholastic’s Interactive Teaching System software. At the conclusion of 
the System 44 training in August 2010 and at the end of the 2010-2011 school year, the 
participating teachers were asked to respond to 10 questions designed to test their 
understanding of the System 44 program components. On average, the teachers answered 
80% of the items correctly in fall 2010 and 81% of the items correctly in spring 2011. 


Exhibit 8 
Reported Use of System 44 Components 
Mean 
Frequency 

System 44 Component Rating SD 

Whole class instruction in System 44 3.58 1.20 
Teaching Guide (bound copy) 3.83 0.72 
Teaching Guide (computer version) 2.83 1.30 
44Book or Decodable Digest (computer version) 3.08 1.40 
SAM reports 4.25 0.75 
Small group instruction: SMART lessons 3.67 0.78 
Individual student support (outside of small group) 4.17 1.00 
SAM worksheets 2.17 0.72 
SAM book expert (to identify appropriate books) 2.08 1.00 
Flip chart 3.08 1.00 
Letter tiles 2.67 0.89 
System 44 audiobooks 3.75 1.20 
Reading Counts quizzes 4.08 0.90 
Sound and Articulation DVD 1.50 0.52 
Conference guides (for System 44 library books) 2.75 1.40 
Scholastic Red routines 3.58 0.90 
Materials not part of System 44 2.42 1.20 


Note. Frequency ratings 1 = rarely or never; 3 = once a week; 5 = every day. Number of 
respondents = 12. 


The teachers also rated their understanding of 5 key System 44 program components in the fall 
and spring (see Exhibit 9). Overall, the teachers reported a greater understanding of how to 
implement the program in spring 2011 than they had in fall 2010, but the differences were not 
statistically significant. In addition, the teachers’ responses varied less in the spring than in the 
fall, a finding that suggests most teachers believed they had a high level of understanding of 
how to use the System 44 program components. 


Instructional Grouping. Interviews with System 44 teachers conducted in spring 2011 
indicated that about half used either the SAM Differentiated Instruction Report or another SAM 
report to group students for small group instruction. By the end of the school year, however, 
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many students had completed System 44 and some class sizes were very small. In those cases 
students used the computer software as a group and then received individual instruction from 
the teacher or worked independently. In one classroom students used the Self-Monitoring 
Checklist to determine whether they needed to use the computer before working independently. 
One teacher reported grouping students based on their ability, and 2 others assigned those 
students who arrived earliest to use the computers during the first half of the class. 


Exhibit 9 
Understanding of Key System 44 Program Components 
Fall Spring 
Item M SD M SD 
| understand... 
How the System 44 software works to individualize 4.25 1.30 4.50 0.67 
instruction and practice for students. 
What to do during small group instruction. 4.25 0.75 4.33 0.65 
How to use SAM to group students for small group 4.42 0.90 4.67 0.49 
instruction. 
When to use the Decodable Digest, the 44Book, and 4.42 0.79 4.83 0.39 
System 44 library books for student practice. 
How to monitor student progress in System 44. 4.25 1.20 4.83 0.39 


Note. Rating scale: 1 = strongly disagree; 5 = strongly agree. Number of respondents = 12. 


Whole Group Instruction. The teachers varied widely in terms of how frequently they 
instructed students as a whole group. Two of the 12 teachers (17%) reported no whole group 
instruction; 8% used whole group instruction less than once a week; 25% did so once a week, 
17% did so 2 or 3 times a week, and 33% did so every day. The reasons cited for using whole 
group instruction infrequently were large class size and a preference for small group instruction. 


SAM Reports. In spring 2011 the interviewer showed each teacher 6 SAM reports and asked 
how often he or she used each report. The teachers most often used the Differentiated 
Instruction Report to identify topics for small group instruction (three fourths reported using it at 
least once a week). The teachers also used the Software Performance Report and the Reading 
Progress report ranging from weekly to every other month. Least frequently used were the 
Response to Intervention Report and the Student Mastery Report. Several teachers commented 
that these reports duplicated information that could be found on other reports. 


Teacher Perceptions of the System 44 Program 


The fall 2010 teacher survey asked respondents to rate the effectiveness of the reading 
intervention they had used the previous year (2009-2010) and their expectations regarding the 
effectiveness of System 44. The spring 2011 survey asked respondents to rate their perceptions 
of the effectiveness of System 44. Exhibit 10 provides the mean ratings for the 3 time points. In 
general, the teachers expected System 44 to be more effective than their prior year’s program in 
5 basic reading skill areas, and these expectations were borne out by their ratings of System 44 
in spring 2011. The difference between the perceived effectiveness of the prior program and 
System 44 with respect to teaching phonemic awareness was statistically significant (p < .05). 
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The teachers also responded to questions regarding the reading instructional practices they 
used with struggling readers both prior to and with System 44. The teachers indicated whether 
each practice listed was central to their reading instruction, a small part of their reading 
instruction, or not part of their reading instruction. The control group teachers responded to the 
same questions regarding the 2010-2011 school year. Exhibit 11 lists the practices for which 
the percentage of teachers reporting that the practice was “central to my instruction” changed by 
at least 25%. Appendix C presents the results for all of the reading instructional practices. 


Exhibit 10 
Perceived Effectiveness of Reading Instruction 

Perceived Expected Perceived 
Effectiveness of Effectiveness of Effectiveness of 

Prior Program System 44 System 44 

(2009-2010) (Fall 2010) (Spring 2011) 

Reading Skill Area M SD M SD M SD 
Phonemic awareness 3.50* 1.10 4.08 1.50 4.50* 0.67 
Phonics 3.73 1.20 4.58 1.20 4.55 0.69 
Fluency 3.91 1.00 4.42 1.20 4.45 0.69 
Vocabulary 4.18 0.60 4.25 1.20 4.18 0.75 
Comprehension 4.09 1.00 4.08 1.20 4.00 1.00 


Note. Rating scale: 1 = Strongly disagree; 5 = Strongly agree. *p < .05. Number of respondents = 12. 


Exhibit 11 
Practices Central to Instruction 

System 44 System 44 Control 

Teachers: Teachers: Teachers: 
Instructional Practice Prior Year 2010-2011 2010-2011 
Provide time in reading block for students to practice 92% 67% 
skills on their own. 
Provide materials for at-home practice of skills 50% 25% 53% 
introduced in class. 
Teach whole class reading lessons. 58% 33% 69% 
Work one-to-one with students on reading. 42% 75% 42% 
Use supplementary reading materials. 58% 27% 
Use books that are easy to decode. 75% 42% 
Use separate intervention materials for some students. 67% 42% 
Use reading software/technology. 92% 49% 
Use tests to determine progress on skills. 92% 58% 
Use diagnostic tests to identify students who need 92% 67% 
reading intervention services. 
Students use graphic organizers to track information. 83% 58% 


Note. System 44 teachers n = 12. Control group teachers n = 33. 
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Of the 11 instructional practices whose importance differed, 9 reflected change in the System 44 
teachers’ instruction between 2009-2010 and 2010-2011 and 5 reflected differences between 
the System 44 teachers and the control group teachers in 2010-2011. The System 44 teachers 
were less likely than the control group teachers to “teach whole class reading lessons” or 
“provide materials for at-home practice,” but more likely than the control group teachers to “work 
one-to-one with students,” “use reading software,” and “use diagnostic tests to identify students 
who need reading intervention services.” Compared to 2009-2010, in 2010-2011 the 

System 44 teachers were more likely to “work one-to-one with students” and less likely to 
“provide materials for at-home practice,” “teach whole class reading lessons,” “use books that 
are easy to decode,” “use tests to determine progress on skills,” and have “students use graphic 
organizers.” 


In spring 2011 the System 44 teachers were asked to discuss the aspects of the program that 
they liked best. The teachers mentioned several software-related features: being able to fast 
track students through skills they had already mastered, the software’s ability to individualize 
instruction according to students’ needs and to revisit skills that students had not yet mastered, 
and the sense of accomplishment and empowerment students gained as they progressed 
through the software. The teachers also liked the SAM reports they could use to track student 
progress and group students for teacher-led instruction on specific skills, the System 44 library 
books, the Decodable Digest, and other program materials. 


Factors That Facilitated Implementation 


The System 44 teachers, principals, and district staff who were interviewed for the evaluation 
identified several factors that facilitated program implementation including district support, 
professional development, software licenses and technology support, teacher dedication, and 
the System 44 materials. 


District Support. Throughout the school year district staff maintained personal communication 
with the participating teachers, principals, and counselors and attended meetings to discuss 
concerns and address challenges. Principals cited support from district staff as a key factor 
facilitating the implementation of System 44. 


Professional Development. Scholastic staff conducted a training in fall 2010, and Scholastic 
staff and district staff provided training and support throughout the school year. Scholastic staff 
provided 3 days of on-site coaching to each teacher, and district staff facilitated meetings for all 
teachers who implemented Scholastic programs. 


Software Licenses and Technology Support. Scholastic and the district maintained a 
sufficient number of working computers to accommodate the students in the program. 
Scholastic provided a sufficient number of software licenses and the district’s technology staff 
provided efficient support to schools to keep the computers and software functioning. 


Teacher Dedication. The teachers selected to implement the System 44 program were 
enthusiastic about the intervention and willing to take on a challenge. Some classrooms had 
teacher aides who provided valuable support—in particular to students who lacked independent 
work skills. Several principals mentioned teacher interest in the System 44 program as one of 
the keys to its success. 


System 44 Materials. The SAM reports enabled the teachers to track student progress and 
identify needs. In addition, the software was able to perform certain tasks that the teachers 
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could not execute with precision with a large number of students. In particular, the software 
identified the specific skills each student needed to improve, implemented the System 44 
program with the utmost fidelity, and provided instruction to every student at his or her skill level. 
Partway through the school year the teachers received Scholastic’s Interactive Teaching 
System, which allowed them to access the print materials associated with System 44 

(e.g., Decodable Digest, 44Book, Teaching Guide) on their laptop computers and project them 
on a screen during teacher-led instruction. Many teachers were very excited about this option 
for delivering content. 


Barriers to Implementation 


The teachers, principals, and district staff also identified several barriers to implementation. Two 
of the greatest challenges occurred at the beginning of the school year and were later resolved: 
scheduling conflicts (particularly at the middle school level) and technology problems. Other 
challenges included classroom teachers’ reluctance to release students for pull-out 
interventions, implementation variation, perceived shortcomings of the System 44 materials, the 
disruption caused by transitioning between activities, the passive consent required for the 
evaluation, and the inconveniences related to participating in the evaluation. 


Scheduling Conflicts. Challenging scheduling conflicts emerged at the beginning of the school 
year. The middle schools in particular experienced difficulty placing the control group students in 
a reading intervention program (some schools placed the control group students in Read 180 
and subsequently had to remove them to comply with the evaluation requirements; others asked 
the district to provide another reading intervention program). Schools that began implementing 
System 44 after the school year had already begun had to shuffle individual students’ 
schedules, and in some schools so few students were assigned to the treatment group that the 
System 44 class could not be included on the master schedule (i.e., it affected the overall 
staffing ratio) and a resource teacher was assigned to teach the class. 


Technology Problems. Teachers faced many technology problems at the beginning of the 
school year, which hindered the initial implementation. For example, older computers would 
freeze and shut down, some of the SAM schoolwide reports caused the computers to shut 
down, the air conditioner noise interfered microphone operation at one school, and microphone 
and headphone problems were prevalent (mostly due to wear and tear). 


Teacher Reluctance to Release Students for Pull-Out Interventions. Some general 
education teachers were reluctant to release their students for the System 44 class. Students 
often arrived late, which interfered with group instruction and made it difficult for the teachers to 
ensure that the students spent the prescribed time using the computer software. Additionally, 
principals expressed concern about pulling students out of content classes such as science and 
social studies, and some parents objected to their children being pulled out of favorite classes. 


Implementation Variation. Most teachers implemented the System 44 program as planned, 
but variations were reported: the teachers at one middle school were constrained by a 
48-minute class period, for part of the year one teacher utilized 3 rotations during a class period 
rather than 2, some classes served students with such a wide range of skills that grouping 
students was difficult, and some teachers supplemented the program materials with worksheets 
of their own design. 


Shortcomings of the System 44 Materials. Some teachers believed that System 44 could be 
improved by providing more supplemental materials for skills practice (particularly for learning 
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disabled students) and more content of interest to middle school students. A few teachers 
mentioned they would have liked more opportunities for students to practice reading fluency, 
and several teachers who desired more support for the System 44 library books added a book 
log worksheet for students to complete for each book they read. In addition, some teachers 
reported that they were not sufficiently familiar with the software to assist students when they 
asked for help. 


Disruption Caused by Transitioning Between Activities. A few teachers noted that the 
transitions between computer work, small group work, and individual work were disruptive. The 
requirement that students spend approximately 20 minutes using the software each day 
exacerbated the problem—some teachers wanted students to be able to move from one activity 
to the next at natural breaking points such as after completing a lesson on the computer, a 
library book, or a worksheet. 


Passive Consent Required for the Evaluation. Some teachers questioned the district’s 
decision to require passive parental consent for student participation in the evaluation. District 
staff believed that active consent was not necessary because students were not participating in 
any activities that were experimental or significantly different from normal instructional activities. 


Inconveniences Related to Participating in the Evaluation. Some principals expressed 
concern that the teachers did not have any input regarding student selection for System 44 and 
that the reading interventions provided to the control group students were not as desirable as 
System 44. Although some principals received no questions or complaints from parents, others 
reported dealing both with disappointed parents who wanted their children in the program and 
parents who did not want their children in the program. 
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Program Impact Findings 


Effects of System 44 on Student Outcomes 


RMC Research used multilevel models to estimate the impact of System 44 on spring 2011 
student outcome scores while controlling for fall 2010 scores, minority status 
(Caucasian/non-Caucasian), and special education status (special education/not special 
education). The same analytic model was run for each of 6 individually administered 
standardized tests: TOSREC, CTOPP Elision, Woodcock-Johnson III Word Identification, 
Woodcock-Johnson III Word Attack, TOWRE Sight Word Efficiency, and TOWRE Phonetic 
Decoding Efficiency. Subsequent analyses examined program impacts on CST English- 
Language Arts scaled scores and on SRI and SPI scores. Appendix D presents the fixed and 
random effects and intraclass correlations for each model in Exhibits D1 through D14. 


Individual Test Outcomes 


The data analysis revealed no significant intervention effects for the overall sample: the 
treatment group students performed similarly to the control group students on each of the 6 
individual tests. Exhibit 12 presents the impact estimates for the overall sample. 


Exhibit 12 
Individual Test Impact Estimates 


Estimated Impact 


Test n Impact (B) SE Effect Size p 
TOSREC 340 -1.88 1.19 -0.11 115 
CTOPP Elision 344 0.54 0.49 0.09 .276 
Woodcock-Johnson III Word Identification 338 0.65 0.84 0.04 442 
Woodcock-Johnson III Word Attack 331 1.08 0.90 0.07 .234 
TOWRE Sight Word Efficiency 344 0.10 0.54 0.01 851 
TOWRE Phonetic Decoding Efficiency 344 -0.19 0.50 -0.02 712 


The standardized effect sizes calculated using Glass’s A method (with the impact estimate as 
the numerator and the control group standard deviation as the denominator) ranged from -0.11 
on the TOSREC to 0.09 on the CTOPP Elision. Exhibit 13 displays descriptive statistics (means, 
standard deviations, and range) for the overall sample. 


RMC Research Corporation @Portland, OR 23 


Exhibit 13 
Individual Test Descriptive Statistics 


Treatment Control 
Test M SD Range M SD Range 
TOSREC® 
Pretest 30.06 15.77 1-90 28.93 16.17 1-69 
Posttest 31.55 16.46 1-80 33.10 16.40 1-85 
CTOPP Elision” 
Pretest 22.38 6.56 0-33 22.58 6.58 0-34 
Posttest 23.97 6.22 0-33 23.53 6.22 0-34 
Woodcock-Johnson III 
Word Identification* 
Pretest 42.45 15.41 1-81 41.61 16.43 1-81 
Posttest 49.82 17.43 1-87 48.59 17.22 1-82 
Woodcock-Johnson III 
Word Attack 
Pretest 44.13 16.11 1-94 43.09 15.48 1-85 
Posttest 49.24 16.17 4-99 47.35 14.53 4-85 
TOWRE Sight Word Efficiency” 
Pretest 62.56 10.10 26-85 62.40 10.79 16-87 
Posttest 67.07 10.49 33-92 66.82 10.74 16-95 
TOWRE Phonetic Decoding 
Efficiency” 
Pretest 30.19 10.41 5-54 30.13 10.95 6-59 
Posttest 33.83 10.64 9-60 33.96 10.71 7-64 


Note. Treatment total n = 173. Control total n = 171. 
“Assessment analyzed using normal curve equivalent scores. >Assessment analyzed using raw scores. 


CST Outcomes 


Analysis of CST pretest and posttest scores revealed no significant differences between 
System 44 students and the control group students on CST gains, 6B = 4.01, p= .344. The 
standardized effect size was 0.09. Exhibit 14 presents descriptive statistics (means, standard 
deviations, and range) for the overall sample. 
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Exhibit 14 
CST Descriptive Statistics 


Treatment Control 
Test M SD Range M SD Range 
Pretest 317.27 34.22 184-404 311.77 35.02 211-402 
Posttest 337.09 43.22 215-455 330.46 43.02 215-425 


Note. Treatment total n = 140. Control total n = 147. Pretest data available for only 287 students; 
posttest data available for 337 students. 


An alternate chi-square analysis examined the percentage of students who achieved 
proficiency—scored at the proficient or advanced performance level—on the CST. Exhibit 15 
presents pretest and posttest proficiency results for the overall sample. Results showed an 
increase in the percentage of students proficient at posttest compared to pretest for both the 
treatment and control groups. Although the percentage increase was slightly higher for the 
treatment group than for the control group, the difference between the treatment and control 
group percentages at posttest were not significant. Appendix E provides further breakdown of 
the percentage of students in the overall sample who performed at each of the 5 performance 
levels (far below basic, below basic, basic, proficient, and advanced) in Exhibit E1. Exhibit E2 
presents the percent of students achieving proficiency by student subgroups. 


Exhibit 15 
Students Achieving Proficiency on CST 


Percent of Students 


2 


Test Treatment Control x p 
Pretest 11 12 0.05 .819 
Posttest 41 32 2.37 124 


Note. Treatment total n = 140. Control total n = 147. Pretest data available for only 
287 students; posttest data available for 337 students. 


SRI and SPI Outcomes 


Multilevel modeling on SRI and SPI outcomes showed positive results for the overall sample. 
The impact was greater for the treatment group students than the control group students on the 
SRI, B = 45.02, p = .008; the effect size on the SRI was 0.21. The treatment group students 
performed similarly to the control group students on 2 of the SPI accuracy tests: SPI Letter 
Name Accuracy and SPI Sight Word Accuracy. However, the treatment group students 
performed significantly better than the control group students on all SPI fluency tests and on 2 
of the accuracy tests. Exhibit 16 presents the impact estimates for the overall sample. 
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Exhibit 16 
SRI and SPI Impact Estimates 


Estimated Impact 


Test n Impact (8) SE Effect Size p 
SRI 332 45.02 16.64 0.21 .008 
SPI Letter Name Accuracy 293 0.27 0.51 0.06 .604 
SPI Sight Word Accuracy 293 0.03 0.30 0.01 .916 
SPI Sight Word Fluency 293 1.74 0.50 0.36 001 
SPI Nonsense Word Accuracy 293 2.00 0.37 0.52 .000 
SPI Nonsense Word Fluency 293 2.84 0.54 0.58 .000 
SPI Overall Accuracy 293 1.99 0.56 0.33 .000 
SPI Overall Fluency 293 4.70 0.88 0.58 .000 


The standardized effect sizes calculated using Glass’s A method ranged from 0.01 to 0.58. The 
effect sizes on tests of fluency were all moderate (0.36, 0.58, and 0.58 for SPI Sight Word 
Fluency, SPI Nonsense Word Fluency, and SPI Overall Fluency, respectively). The effect sizes 
on 2 of the accuracy tests—SPI Nonsense Word Accuracy and SPI Overall Accuracy—were 
also moderate (0.52 and 0.33, respectively). Exhibit 17 displays descriptive statistics (means, 
standard deviations, and range) for the overall sample. 


Exhibit 17 
SRI and SPI Descriptive Statistics 
Treatment Control 

Test M SD Range M SD Range 
SRI 

Pretest 404.40 174.96 0-907 401.56 202.14 0-873 

Posttest 585.46 207.77 0-1,040 541.45 218.75 0-1,041 
SPI Letter Name Accuracy 

Pretest 96.19 11.52 18-100 96.11 12.15 0-100 

Posttest 98.47 3.85 82-100 98.27 4.88 64-100 
SPI Sight Word Accuracy 

Pretest 21.32 3.80 9-28 21.23 4.08 9-30 

Posttest 23.95 3.16 15-30 23.92 3.47 14-30 
SPI Sight Word Fluency 

Pretest 6.54 3.34 0-14 7.01 3.17 0-14 

Posttest 11.01 4.62 3-23 9.6 4.77 0-24 


Exhibit continues 


26 Evaluation of System 44 


Exhibit 17 (continued) 


Treatment Control 

Test M SD Range M SD Range 
SPI Nonsense Word Accuracy 

Pretest 20.82 3.94 10-29 20.55 3.73 9-30 

Posttest 25.05 3.17 14-30 22.99 3.82 10-30 
SPI Nonsense Word Fluency 

Pretest 6.97 3.47 0-18 7.14 3.45 0-14 

Posttest 12.02 5.39 1-22 9.42 4.86 0-21 
SPI Overall Accuracy 

Pretest 42.14 6.53 25-57 41.79 6.22 21-56 

Posttest 49.00 5.55 34-60 46.9 6.08 27-59 
SPI Overall Fluency 

Pretest 13.51 5.23 1-23 14.15 5.20 1-22 

Posttest 23.03 8.87 4-43 19.02 8.17 0-38 


Note. One school assessed only a subset of students at posttest. For that school SRI posttest data were available 
for some treatment and control students and were retained in the analyses. SPI data were available for a subset of 
treatment students only and were removed from the analyses. SRI Treatment total n = 173. Control total n = 159. 
SPI Treatment total n = 147. Control total n = 146. 


Differential Effects of System 44 on Subgroups of Students 


The hierarchical linear models conducted for each of the 6 individual reading test outcomes and 
the CST, SRI, and SPI outcomes consistently revealed main effects for special education (see 
Appendix D). Specifically, special education students scored significantly lower than non-special 
education students on the TOSREC, CTOPP Elision, Woodcock-Johnson III Word Identification, 
Woodcock-Johnson III Word Attack, CST, SRI, SPI Sight Word Fluency, SP! Nonsense Word 
Accuracy, SPI Nonsense Word Fluency, and SPI Overall Fluency tests. Exhibits 18-20 present 
the pretest and posttest means and standard deviations for special education and non-special 
education students for each test. Exhibits F1 through F9 in Appendix F present mean pretest, 
posttest, and gain scores on individual reading test, SRI, and SP! Overall Accuracy and Fluency 
outcomes for additional subgroups of students. 


RMC Research Corporation @Portland, OR 27 


Exhibit 18 
Individual Test Descriptive Statistics for Special Education Students 


Special Education Non-Special Education 

Test M SD M SD 
TOSREC® 

Pretest 20.91 16.38 32.98 14.42 

Posttest 22.78 16.09 36.18 14.94 
CTOPP Elision” 

Pretest 19.44 7.15 23.76 5.85 

Posttest 20.94 7.04 24.93 5.43 
Woodcock-Johnson III Word Identification 

Pretest 29.48 15.95 47.15 12.75 

Posttest 35.64 18.67 54.75 13.23 
Woodcock-Johnson III Word Attack* 

Pretest 33.22 16.05 48.05 13.44 

Posttest 38.24 15.26 52.59 13.31 
TOWRE Sight Word Efficiency” 

Pretest 57.17 12.82 64.72 8.31 

Posttest 61.80 12.88 69.12 8.63 
TOWRE Phonetic Decoding? Efficiency 

Pretest 25.50 11.13 32.12 9.85 

Posttest 29.76 11.35 35.64 9.88 
CST° 

Pretest 302.74 37.66 318.23 32.87 

Posttest 331.10 52.19 334.53 39.93 


Note. Special education total n = 102. Non-special education total n = 242. Both treatment and control included. 
“Assessment analyzed using normal curve equivalent scores. >Assessment analyzed using raw scores “Assessment 
analyzed using scaled scores. 
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Exhibit 19 presents the means and standard deviations for special education and non-special 
education students on the CST. Although the results indicated that the special education 
students performed lower than the non-special education students at pretest, the special 
education and non-special education students performed similarly at posttest. 


Exhibit 19 
CST Descriptive Statistics for Special Education Students 
Special Education Non-Special Education 
Test M SD M SD 
Pretest 302.74 37.66 318.23 32.87 
Posttest 331.10 52.19 334.53 39.93 


Notes. Special education total n = 70. Non-special education total n = 217. Both 
treatment and control group included. CST analyzed using scaled scores. 


To determine whether these effects were a result of System 44— that is, did special education 
System 44 students make significantly greater gains than special education control group 
students?—the moderating effects of special education status on treatment were analyzed. The 
results revealed no significant moderating effects. Similar tests of moderating effects were 
conducted for SRI and SPI outcomes and again no moderating effects of System 44 and special 
education status were found for any of the SRI and SPI outcomes. Thus the observed effects of 
System 44 were equivalent for special education and non-special education students (see 
Exhibit 20). 


Although the most consistent demographic subgroup main effect observed across tests was 
among special education students, the main effect of sex was significant for the TOSREC, on 
which girls scored higher than boys, and Woodcock-Johnson III Word Attack, on which boys 
scored higher than girls. Nevertheless, overall the effects of System 44 did not vary by 
demographic subgroup. 


Exhibit 20 
SRI and SPI Descriptive Statistics for Special Education Students 
Special Education Non-Special Education 

Test M SD M SD 
SRI? 

Pretest 367.92 202.94 416.50 180.85 

Posttest 484.54 246.60 594.99 191.89 
SPI Letter Name Accuracy” 

Pretest 98.42 4.04 95.38 13.39 

Posttest 98.18 4.21 98.44 4.45 
SPI Sight Word Accuracy” 

Pretest 20.32 4.48 21.60 3.69 

Posttest 23.15 3.78 24.20 3.10 


Exhibit continues 
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Exhibit 20 (continued) 


Special Education Non-Special Education 

Test M SD M SD 
SPI Sight Word Fluency” 

Pretest 6.65 3.58 6.81 3.15 

Posttest 9.89 5.17 10.45 4.59 
SPI Nonsense Word Accuracy” 

Pretest 19.61 3.99 21.05 3.71 

Posttest 22.77 4.01 24.44 3.43 
SPI Nonsense Word Fluency? 

Pretest 5.96 3.29 7.43 3.43 

Posttest 9.32 5.55 11.20 5.12 
SPI Overall Accuracy” 

Pretest 39.93 6.83 42.65 6.07 

Posttest 45.92 6.79 48.64 5.42 
SPI Overall Fluency? 

Pretest 12.61 5.49 14.24 5.06 

Posttest 19.22 9.25 21.65 8.51 


*SRI special education total n = 92. SRI non-special education total n = 240. °SPI special education total 
n=74. SPI non-special education total n = 219. 


Association Between Program Implementation and System 44 Gains 


The results of the implementation analyses showed some variation in implementation across 
schools. To explore the possible effects of implementation on System 44 impact, the evaluation 
team conducted nonexperimental analyses to address the following question: To what extent is 
program implementation associated with treatment impact? 


The data collected from the teacher surveys and classroom observations were used to develop 
5 teacher-level variables (aggregated by school level) used to assess System 44 program 
implementation in relation to test outcome gains: teacher’s prior use of System 44, total number 
of times the teacher used System 44 materials during small group instruction, average number 
of students using System 44 paperback books or the Decodable Digest during independent 
work time, instructional management, and instructional delivery. Prior use of System 44 (yes/no) 
was collected from the fall 2010 teacher surveys. The variable measuring the number of times 
the teacher used System 44 materials was a sum across fall, winter, and spring classroom 
observations. The average number of students using System 44 paperback books or the 
Decodable Digest during independent work time was an average across the 3 classroom 
observation time points. The instructional management variable comprised 2 measures (used 
flexible groups, used differentiated support) that were averaged across fall, winter, and spring. 
Similarly, the instructional delivery variable comprised 2 measures (executed the lesson well, 
actively tried to engage students) that were averaged across the 3 observation points. 
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Multilevel models identical to those used to measure impact—but with the 5 teacher-level 
variables (aggregated by school level) added to Level 2 of the model—were used to estimate 
the effects of implementation on the impact of the intervention on spring 2011 test outcomes. 
The addition of the teacher-level variables to the model revealed that none of the 5 variables— 
teacher's prior use of System 44, total number of times the teacher used System 44 materials 
during small group instruction, average number of students using System 44 paperback books 
or the Decodable Digest during independent work time, instructional management, and 
instructional delivery—were associated with treatment effects. 


Association Between Software Usage and System 44 Gains 


Because student use of the System 44 software is an essential component of the program, the 
evaluation team examined how students’ System 44 software usage was related to improved 
individual test scores, CST scores, and SRI and SPI scores by analyzing the outcome gains in 
relation to the following factors: program exit date and total number of series topics completed. 
Exhibits G1 and G2 in Appendix G present characteristics of students by exit date, total number 
of series topics completed, and Exhibits H1 and H2 in Appendix H present baseline descriptive 
data. Additionally, because a student’s initial decoding status on the SPI determines the starting 
series in System 44, Exhibit G3 presents characteristics of students by initial decoding status, 
and Exhibit H3 presents baseline descriptive data by initial decoding status. Appendix | presents 
System 44 software usage descriptive data by student subgroups. 


Program Exit Date 


Although the System 44 program was intended to be implemented for a full year, 26% of the 
treatment group students exited the program in winter (i.e., prior to March 2011), an additional 
33% exited the program in spring (i.e., March or April 2011), and the remaining 41% stayed in 
the System 44 program through the end of the school year (i.e., May or June 2011). To examine 
differences in the characteristics of the students who exited the program at each time point, the 
evaluation team conducted baseline equivalence tests between the 3 groups. Exhibit H1 in 
Appendix H presents baseline descriptive data for the winter, spring, and end-of-year exits. 


The students who stayed in the System 44 program through the end of the year had significantly 
lower pretest scores than the students who exited the program in winter or spring on all tests 
except SPI Letter Name Accuracy. The only demographic characteristic that differed across the 
3 groups was special education status—that is, those students who stayed in the program 
through the end of the year were significantly more likely to be in special education than the 
students who exited in the spring. For some tests the significant pretest score discrepancies 
were between the end-of-year exits and the other 2 groups. For example, there was no 
difference between the students who exited in winter or spring on mean pretest CST scores 

(Ms = 322.51 and 324.42, respectively), but the differences between those students who exited 
in winter or spring and the students who exited at the end of the year (M = 300.90) were 
significant. On other tests the students who exited in winter scored significantly higher at pretest 
than the students who exited in spring, and the students who exited in spring scored 
significantly higher at pretest than the students who exited at the end of the year. The average 
pretest SRI Lexile scores for students exiting in winter, spring, and end of year, for example, 
were 498.91, 408.14, and 343.66, respectively. 


To address whether overall test score gains were significant and whether the timing of program 
exit factored into students’ gains, repeated measures analysis of variance (ANOVA) models 
were used to assess differences between program exit date categories (i.e., winter, spring, end 
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of year) and change in outcomes. The analyses revealed that overall gains in scores were 
significant for all tests except the TOSREC. Paired t-test analyses were conducted to assess 
whether the lack of gains observed on the TOSREC varied across subgroups. The results 
showed that students who exited in the spring made marginally significant gains on the 
TOSREC, ¢(56) = 1.78, p = .08, but the students who exited in the winter and the end of the year 
made essentially no gains. 


In addition, the results of the repeated measures ANOVA analyses revealed that timing of 
program exit did not factor into students’ gains on any test except one—that is, the gains were 
similar for the winter, spring, and end-of-year subgroups. The one exception was the SPI 
Overall Fluency test, on which students who exited the program in winter or spring made 
significantly greater gains than did students who exited at the end of the year, F(2, 165) = 3.60, 
p<.05. 


Topic Completion 


The System 44 students were expected to complete all 160 topics covered in the 25 series that 
compose the software component of the program. The analysis conducted by the evaluation 
team revealed that approximately half of the System 44 students completed all 160 topics 
(average number of topics completed: 133). The students were categorized by the number of 
topics completed: fewer than 100 topics (n = 48), between 100 and 159 topics (n = 37), and all 
160 topics (n = 92). Repeated measures ANOVA models were used to assess differences 
between the 3 topic completion groups and change in outcomes. 


The ANOVA results showed significant differential gains between the 3 groups on the following 
tests: Woodcock-Johnson III Word Identification (p < .05), SPI Sight Word Fluency (p < .001), 
SPI Nonsense Word Fluency (p < .001), and SPI Overall Fluency (p < .001). Specifically, those 
students who completed fewer than 100 topics showed significantly less gain on these 4 tests 
than the students who completed between 100 and 159 topics and the students who completed 
all 160 topics (see Exhibits 21-24). In addition, on 2 of the 4 tests for which differential gains 
were observed (SPI Sight Word Fluency and SPI Overall Fluency), the students who completed 
all 160 topics showed significantly greater gains than the students who completed between 100 
and 159 topics. Additional analyses examined differences between these 3 groups on baseline 
characteristics (see Exhibit H2). Those students who completed fewer than 100 topics had 
significantly lower pretest scores than the students who completed more than 100 topics on all 
tests with the exception of SPI Letter Name Accuracy. 
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Exhibit 21 


Woodcock-Johnson Ill Word ID Gains by Number of Topics Completed 


Normal Curve Equivalent Scores 


Fall 2010 Spring 2011 


Pre- and Posttest 


=—=@=—= Less than 100 Topics 


—M=— Between 100 and 159 
Topics 
eede* All 160 Topics 


Note. Scores range from 1-100. Less than 100 Topics n = 43, 


Between 100 and 159 Topics n = 37, All 160 Topics n = 92. 


—@=— Less than 100 Topics 


Exhibit 22 
Sight Word Fluency Gains by Number of Topics Completed 
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Exhibit 23 


SPI Nonsense Word Fluency Gains by Number of Topics Completed 


Scaled Score 


Scaled Score 
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Note. Scores range from 1-30. Less than 100 Topics n = 43, 
Between 100 and 159 Topics n = 37, All 160 Topics n = 92. 


Exhibit 24 
SPI Overall Fluency Gains by Number of Topics Completed 
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Note. Scores range from 1-60. Less than 100 Topics n = 43, 
Between 100 and 159 Topics n = 37, All 160 Topics n = 92. 
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Evaluation of System 44 


Conclusion and Recommendations 


Of the 425 students in Grades 4—8 who met the eligibility criteria for System 44, 216 were 
randomly assigned to the treatment group and 209 were randomly assigned to the control 
group. The groups were equivalent in terms of sex, eligibility for free or reduced-price meals, 
English proficiency, special education status, ethnicity, and CST scores. Of the randomly 
assigned students, 172 treatment and 173 control group students received the allocated 
intervention or control group condition as planned. 


Local testers individually administered a battery of 6 standardized reading tests to all treatment 
and control group students in fall 2010 and spring 2011. In addition, the evaluation team 
assessed change on the CST English-Language Arts subtest, the SRI, and the SPI. The 
evaluation team also administered surveys to the System 44 teachers in fall 2010 and spring 
2011 and to a sample of control group teachers in spring 2011; observed System 44 classes 3 
times during the 2010-2011 school year (November, February, and April); and interviewed 
System 44 teachers, principals, and district staff in spring 2011. 


The evaluation team analyzed the data collected to address 5 research questions: 


1. What contextual factors are involved in the implementation of System 44 (i.e., factors 
that promote or hinder successful implementation of the program)? 


Although the control group teachers had more teaching experience and advanced degrees, the 
System 44 teachers were more likely to have Education Specialist or Reading Specialist 
certification and to have prior experience teaching System 44 or Read 180. Overall, the 
treatment group teachers gave the System 44 professional development high ratings, although 
the mean ratings were somewhat lower in spring 2011 than in fall 2010. They were pleased with 
the quantity and quality of the training and believed that it prepared them to use the System 44 
program in their classrooms. In response to 10 questions designed to test their knowledge of 
System 44 program components, on average the teachers answered correctly 80% of the 
questions in fall 2010 and 81% of the questions in spring 2011. 


The evaluation team assessed fidelity of System 44 implementation through classroom 
observations and teacher self-report (surveys and interviews). The adequacy of classroom 
setup improved from 58% of the classrooms in the fall to 75% in the winter and 100% in the 
spring. Minutes of instruction were consistent across all 3 observation points: 9 of the 12 
classrooms provided at least 55 minutes of System 44 instruction daily and the remaining 
classrooms provided 45 to 49 minutes of instruction daily. One third of the teachers reported 
providing whole group instruction daily, 17% did so 2 or 3 times a week, and the remaining 50% 
did so once a week or less. During teacher-led small group instruction, the observers most 
frequently noted the use of the System 44 flip chart, letter tiles, and the Decodable Digest. 
During independent work time, the observers most frequently noted the use of the 44Book, 
System 44 paperback books, and the Decodable Digest. 


The observers rated instructional management and instructional delivery lowest in the fall and 
highest in the winter. Ratings on 2 key instructional management items were lowest in the fall 
(“use flexible groups for students based on instructional needs” and “differentiate support based 
on students’ needs”) but were higher in the winter. At the time of the spring interviews, 75% of 
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the teachers reported having used the SAM Differentiated Instruction Report at least once a 
week to identify topics for small group instruction. 


The System 44 teachers, school administrators, and district staff also identified several barriers 
to implementation. Two of the greatest challenges occurred at the beginning of the school year 
and were later resolved: scheduling conflicts (particularly at the middle school level) and 
technology problems. Other challenges included teacher reluctance to release students for pull— 
out interventions, implementation variations, perceived shortcomings of the System 44 
materials, disruption caused by transitioning between activities, passive consent required for the 
evaluation, and inconveniences related to participating in the evaluation. 


2. What are teachers’ perceptions of the program? 


Prior to the start of the school year, the teachers expected System 44 to be more effective than 
the reading intervention they had used the prior year; at the end of the school year the teachers 
gave System 44 significantly higher ratings in terms of its perceived effectiveness for teaching 
phonemic awareness than the prior programs. Effectiveness ratings for phonics and fluency 
were also higher for System 44, but the differences were not statistically significant. 
Effectiveness ratings for vocabulary and comprehension were essentially the same for 

System 44 and the prior programs. When asked which features of System 44 they liked best, 
the teachers mentioned several software-related features: being able to fast track students 
through skills they had already mastered, the software’s ability to individualize instruction 
according to students’ needs and to revisit skills that students had not yet mastered, and the 
sense of accomplishment and empowerment students gained as they progressed through the 
software. The teachers also liked the SAM reports they could use to track student progress and 
group students for teacher-led instruction on specific skills, and the program materials. 


3. What are the effects of System 44 on student outcomes? Specifically, how do 
changes in word reading accuracy, fluency, and comprehension achieved by 
System 44 students compare to changes achieved by a control group? 


The data analyses revealed intervention effects on the SRI, the 3 SPI fluency tests, and 2 of the 
SPI accuracy tests (Nonsense Word and Overall Accuracy). No significant effects were found 
on any of the 6 individual reading tests, the CST, or SPI Letter Name or Sight Word Accuracy. 
Although intervention effects were not observed on individual reading tests, analyses revealed 
significant main effects on all individual reading tests—that is, treatment and control group 
students alike made significant gains between pretest and posttest on all outcomes. 


4. How does System 44 differentially affect subgroups of students? Specifically, how do 
changes in word reading accuracy, fluency, and comprehension achieved by specific 
subgroups of System 44 students (English language learners, students in special 
education, students receiving free or reduced-price meals, and ethnic minority 
students) compare to changes achieved by subgroups of control group students? 


Analyses on subgroups of students revealed that special education students scored significantly 
lower on many of the outcome measures than non-special education students (TOSREC, 
CTOPP Elision, W-J Word Identification, W-J Word Attack, CST, SRI, and SPI Sight Word 
Fluency, Nonsense Word Fluency, Overall Fluency, and Nonsense Word Accuracy). However, 
there were no moderating effects of System 44 and special education status on any of the 
outcomes—that is, the observed effects of System 44 were equivalent for special education and 
non-special education students. 
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5. What is the association between System 44 effects and program implementation—are 
changes in System 44 participants’ word reading accuracy, fluency, and 
comprehension associated with variation in program implementation? 


The evaluation team used data collected from teacher surveys and classroom visits to develop 
5 teacher-level variables for assessing the extent to which program implementation is 
associated with program impact: teacher’s prior use of System 44, total number of times the 
teacher used System 44 materials during small group instruction, average number of students 
using System 44 paperback books or the Decodable Digest during independent work time, 
instructional management, and instructional delivery. None of these implementation variables 
were associated with treatment effects. 


The evaluation team analyzed gains on each outcome measure in relation to program exit date 
and total number of topics completed. Students were grouped into 3 exit date categories: winter 
(prior to March 2011), spring (March or April 2011), and end of year (May or June 2011). Not 
surprisingly, students who stayed in the System 44 program through the end of the school year 
had significantly lower pretest scores than students who exited in the winter or spring. Students 
in the end-of-year group were also significantly more likely to be in special education than 
students in the spring exit group. Repeated measures ANOVA analyses revealed that exit date 
did not factor into students’ gains except for one assessment—the SPI Overall Fluency subtest, 
in which students who exited in winter or spring made significantly greater gains than students 
who stayed in the System 44 program through the end of the school year. 


Approximately half of the System 44 students completed all 160 topics, and the average number 
of topics completed was 133. Students were grouped by number of topics completed (fewer 
than 100 topics, between 100 and 159 topics, and all 160 topics), and baseline equivalence 
tests showed significant differences between the 3 groups on all but 1 pretest measure. 
Repeated measures ANOVA analyses examined differences between 3 topic completion 
groups:. Analyses revealed that there were differential effects between groups on 4 outcomes. 
Specifically, students who completed fewer than 100 topics made significantly less gain on W-J 
Word Identification, SPI Sight Word Fluency, SPI Nonsense Word Fluency, and SPI Overall 
Fluency than students in the other completion groups. In addition, on 2 of these tests (SPI Sight 
Word Fluency and SPI Overall Fluency) the students who completed all 160 topics showed 
significantly greater gains than the students who completed between 100 and 159 topics. 


Recommendations 


The evaluation team makes several recommendations based on the findings summarized in this 
report. 


1. Although teachers reported feeling well-prepared to teach System 44 after their initial 
training, observed variations in program implementation suggest that more in-depth 
training might help facilitate the implementation of System 44 in the classroom. For 
example, based on classroom observations, teachers tended to implement flexible 
groups and differentiated support less in the fall than in winter and spring. In addition, 
certain program materials such as System 44 audiobooks appeared to be under-utilized 
year-round. RMC Research recommends modifying the initial training session to include 
an additional half day or full day to provide more in-depth opportunities for teachers to 
practice using SAM reports to identify students in need of differentiated instruction and to 
group students. During this additional time, more time could also be spent demonstrating 
how specific System 44 resources might be used to address some of those student 
needs. 
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There was a fair amount of variation between teachers in their management of rotations 
during classroom visits. A few teachers implemented a system in which students 
managed their own rotations between computers and other activities (e.g., students 
moved off the computer at the end of a topic to work on associated print materials), 
whereas other teachers adhered to a schedule and stopped computer work at a 
particular point in time and asked students to rotate. The latter strategy ensured at least 
25 minutes on the computer software, but took more time for the rotation because 
students had to come to a stopping point before they could exit the software, which left 
the next group having to wait for a free computer. Thus the evaluation team 
recommends that more specific direction about managing rotations between computers, 
small groups, and independent work be emphasized in the training. 


During the end of year interviews, teachers reported not being familiar enough with the 
System 44 software to address certain questions that the students had. Because 
teachers’ exposure to the software itself is fairly limited, it might be useful for teachers to 
have the opportunity to explore the program on their own. Thus RMC Research 
recommends that System 44 teachers have a teacher-license to the software program 
that allows them to explore its different features as needed. 


A key finding of this study based on groupings of students by exit date and number of 
topics completed was that certain groups of students who participated in System 44 
looked very different at baseline. Some of the students lacked only specific skill sets and 
progressed through the 160 topic areas quickly, whereas other students lacked more 
fundamental skills and took much longer to progress through the program. Based on exit 
dates, 3 distinct groups were evident: students requiring approximately 1 semester to 
complete the program, students requiring 1 year to complete the program, and students 
who were still only part way through the program after 1 school year. Although all groups 
made progress, the group that stayed in the program through the end of the year scored 
significantly lower at posttest than students who exited the program in winter or spring. 
Grouping by number of topics completed similarly produced groups that differed 
significantly at baseline. Based on these findings, the evaluation team recommends that 
the System 44 model of instruction and duration of instruction be tailored to the baseline 
characteristics of the students. A student’s performance on the SRI, for example, could 
be used to help inform teachers about the type of model that would be most effective for 
that student—a semester model, one-year model, or multi-year model, for example. 


Findings from this study demonstrated variation among the tests used to measure 
impact. Most of the individual tests selected for measuring program impact did not reveal 
any differences between the treatment and control groups, whereas other measures 
(SRI and SPI) showed significant System 44 impacts. Given these discrepancies, the 
evaluation team recommends that further research be conducted to identify measures 
that may be more sensitive to the skills taught in the System 44 program. 


Evaluation of System 44 


Appendix A 
Data Collection Instruments 


System 44 Teacher Survey 


[Ui aals)ts ma Olalli(svemexe acele) mB) iiq(er! 


August 2010 


About This Survey 


This survey contains questions about your background and questions about the 

System 44 program that you will be implementing this school year. Your responses are 
important in helping us understand which factors contribute to the success of the program 
and what issues need to be addressed. 


Please write your answers directly on the survey by checking the appropriate boxes or by 
writing your response in the space provided. Your individual responses will be kept 
confidential and will not be shared with your principal or other school or district personnel. 
Summary data from this survey will be shared with Scholastic staff to assist in planning 
for System 44 implementation. 


We expect it will take approximately 15 minutes to complete the survey. 


Thank you very much for your help. 


Prepared by: 
RMC Research Corporation 
111 SW Columbia Street 
Suite 1200 

RESEARCH Portland, OR 97201 


Teacher Background 


1. Are you female or male? 
O, Female 
Oo, Male 
2. How many years have you worked as a full-time teacher in public schools ? Write in the number 


of years below. Count part of a year as one year. 

Number of years 
a. Total number of years as a teacher (do not include student teaching) 
b. Total number of years as a teacher in Murrieta Unified School District 


c. Total number of years as a teacher at your current school 


3. What is the highest degree you have obtained as of September 2010? Mark (X) only one box. 
CO, Bachelor's (B.A., B.S., B.E., etc.) 
1, Master’s degree (M.A., M.A.T., M.B.A., M.Ed., M.S., etc.) 
O, Education specialist or certification at least one year beyond master’s level 


(1, Doctorate or professional degree (Ph.D., Ed.D., M.D., L.L.B., J.D., D.D.S.) 


O.; Other (please specify: ) 
4. Which of the following describes the teaching certificate(s) you currently hold in California? 
Mark (X) all that apply. 
1, Single subject O, Reading Specialist 
, Multiple subject Oj, Administrative Services 
O, Preliminary 1, District Intern 
4 Professional Clear ©; CLAD or BCLAD (Bilingual, Crosscultural, 


Language and Academic Development 
O. Education Specialist Instruction ere P ) 


5, | have used System 44 in my classroom prior to this year. 
1, Yes Ol, No 

6. | have used Read 7180 in my classroom prior to this year. 
O, Yes O1, No 
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Professional Development 


re Have you had training or support (e.g., in-classroom coaching, online support) provided by a 
Scholastic representative or other experienced System 44 user prior to today’s training? 
Mark (X) all that apply. 


O. Have not received any training in System 44 prior to today’s training 
O., Attended a similar System 44 training held in a prior year 
O. Received other support for System 44 from a Scholastic representative in a prior year 


1, Received other support from another experienced System 44 user in a prior year 


O. Other (please specify: ) 
8. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 training activities provided today. Mark (X) only one box per row. 
Strongly Strongly 
disagree agree 


a. The teacher training on System 44 prepared me to use 0, Oo, 0, Oo, Oo, 
the program in my classroom. 


b. lam pleased with the amount of System 44 0, fale 0, Oo, Oo, 
professional development | received. 


c. lam pleased with the quality of System 44 0, Oo. 0, Oo, Oo, 
professional development | received. 


Instructional Practices 


9. Please indicate which of the following teaching strategies and materials you used with your 
struggling readers last year (2009-2010). Mark (X) only one box per row. 


If you did not teach last year but taught in previous years, please describe your use of these 
teaching strategies and materials when you last taught. If this is your first year teaching, please 
skip to Question 9. 


Central to Small part of Not Part of 
my reading my reading my reading 


instruction instruction instruction 
Instruction 

a. Provide time in reading block for students to 0, O., O, 
practice skills on their own. 

b. Provide materials for at-home practice of skills 0, O. Oo, 
introduced in class. 

c. Provide extra reading instructional time for 0, Oo, 0, 
struggling readers. 

d. Include writing opportunities in reading instruction. 1, 1, O, 

e. Build spelling practice into reading instruction. O, Oo, 0, 

f. Develop reading skills using science and social O, Oo, 0, 


studies texts. 
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Central to Small part of Not Part of 
my reading my reading my reading 


instruction instruction instruction 
Grouping 
g. Teach whole class reading lessons. O, 0, 0, 
h. Work one-to-one with students on reading. 1, 1, 0, 
i. Work with small groups of students. O, Oo, 0, 
j. Group students based on skill levels. 0, 1, O; 
k. Group students based on need for additional 0, O. Oo, 
instruction in specific, targeted skills. 
|. Group students based on mixed abilities (pairs or 0, O. O, 
cooperative groups). 
Reading Materials 
m. Use core reading series. O, 0, 0, 
n. Use supplementary reading materials. OF, O me 
o. Use trade books. O, Oo, Oo, 
p. Use books that are easy to decode. 0, 1, ime 
q. Use separate intervention materials for some 0, O. OH, 
students. 
r. Use reading software/technology. O, 0, 0, 
s. Use teacher-made materials. 0, OO, mp 
Assessments 
t. Use test results to organize instructional groups. OF, Oo, 0, 
u. Use informal reading inventories. OF, Oo, 0, 
v. Conduct miscue analysis, analyzing errors 0, 1, OH, 
students make while reading aloud. 
w. Use tests to determine progress on skills. O, Oo, 0, 
x. Use diagnostic tests to identify students who need 0, 1. O, 


reading intervention services. 
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10. Please indicate which of the following reading instructional activities you used with your 
struggling readers last year (2009-2010). Mark (X) only one box per row. 


If you did not teach last year but taught in previous years, please describe your use of these 
reading instructional activities when you last taught. If this is your first year teaching, please skip 
to Question 9. 


Central to Small part of Not Part of 
my reading my reading my reading 


instruction instruction instruction 
Reading Text 
a. Students reread familiar text. 0, em 0, 
b. Students confirm or revise predictions after OF, Oo, 0, 
reading. 
c. Students generate their own questions about text 0, O., OH, 
material. 
d. Students identify their comprehension break-downs 0, Oo, 0, 
and use fix-up strategies with a partner. 
e. Students orally summarize main events in stories 0, Oo, 5 
and informational texts. 
f. Students use graphic and semantic organizers to 0, O., OH, 
track information. 
Work With Sounds and Words 
g. lIteach specific strategies for decoding unfamiliar 0, im O, 
words. 
h. |teach decoding/phonics skills while reading OF, Oo, 0, 
stories. 
i. Students practice reading high frequency words for 0, 1, O, 
automaticity. 
j. Students use knowledge of root words, prefixes, O, Oo, 0, 
and suffixes to decode new words. 
k. Students work with prefixes and suffixes to change O, Oo, 0, 
the meaning of words. 
|. Students use context clues to identify unknown 0, O. OH, 
words. 
m. | discuss new and unusual words before reading. O, Oo, 0, 
Other Techniques 
n. Students answer questions in writing after reading 0, Oo, 0, 
stories. 
0. Students select books from the library for 0, 0, OH, 
independent reading. 
p. Students are given time to read on their own for O, Oo, 0, 


enjoyment. 
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i a Please rate the extent to which you agree or disagree with the following statements. Mark (X) 
only one box per row. 


My struggling students last year received enough 


instruction and practice in the following areas to Strongly Strongly 
make sufficient gains in reading. disagree agree 
a. phonemic awareness of Gs OBO: Gs DB: 
b. phonics oy 3 GF 2s. Gs 
c. fluency ch ie. “Ee we “eke 
d. vocabulary Gy ‘Ei <a jh: “Ge 
e. comprehension on, OF OF % Oo; 
I think System 44 will be effective this Strongly Strongly 
year in helping my students with... disagree agree 
a. phonemic awareness. on, OF OF &% OO; 
b. phonics. Ely <e ° is: 26h, - .ak 
c. fluency. Gl ‘Eh: “a “he “EE 
d. vocabulary. on f% Oo % OO; 
€. comprehension. oo 8 5 36; -Ge 
System 44 Concepts 


The following questions are intended to capture your current understanding of the System 44 program 
based on either prior knowledge of the program or what you learned in the training today. This 
information will help us to understand areas that the training did or did not address, and will help guide 
future training sessions. 


12. Which of the following is NOT a skill that is measured by the Scholastic Phonics Inventory? 
Mark (X) only one box. 


1, Identify sight words 
O, Recognize letter names 
Identify nonsense words 


1, Improve phonological awareness 
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13. Which of the following System 44 program components is designed to provide students with 
written practice in applying the skills they are learning in the software? Mark (X) all that apply. 


O), Decodable Digest 
1, 44Book 
1. Flip chart 
OO, SAM Practice Pages 
14. Which of the following System 44 program components is designed to provide reading fluency 
practice for students? Mark (X) all that apply. 
O. Decodable Digest 
1, System 44 Library Books 
O1. System 44 Audiobooks 


O, Success Strand 


15; Scholastic Reading Counts! is ... Mark (X) only one box. 
Oj, the System 44 adaptive software that continuously collects data on student performance. 
©. aset of quizzes that measure comprehension of each System 44 paperback book. 
3 aset of progress monitoring tools given at the end of each System 44 series. 
©, aset of motivational videos that students can play when they have successfully completed 
a System 44 series of lessons. 
16. Which of the following is NOT a feature of System 44? Mark (X) only one box. 
1, Using SAM reports for instructional grouping 
©. Teacher-led instruction in small groups 
3 Library books that reinforce specific phonics skills 


1. Successful completion of every software lesson 


17. Phonemic awareness is . . . Mark (X) all that apply. 
Othe foundation for phonics instruction. 
1, the ability to hear and manipulate individual sounds in spoken words. 
O. the same as phonological awareness. 


(1, the recognition that sentences consist of separate words. 


RMC Research Corporation Portland, OR System 44 Study Design 


18. The Scholastic Achievement Manager (SAM) can .. . Mark (X) all that apply. 
help teachers form instructional groups for students who need work on the same skills. 
1, provide detailed, diagnostic information about student strengths and weaknesses. 
©. allow the teacher to monitor students’ ongoing progress on the software. 
4 provide downloadable assessments for various reading skills. 


19. Select the description from the right column that matches each of the four System 44 Zones 
listed on the left and enter the matched number in the box: 


a. Fluency Zone 3) 1 Explicit instruction in the 44 sounds of the English 
language and the letters that make each sound 

b. Spelling Zone i =-| 2 Reading of decodable text with a focus on 
comprehension 

c. Word Zone [ 3 Instruction and practice in specific spelling rules 

d. Smart Zone [_} 4 Modeling and practice in blending sounds 


20. Select the description from the right column that matches each of the four System 44 strands of 
instruction listed on the left and enter the matched number in the box: 


a. Success ie 1 Builds automatic recognition of high frequency 
words 

b. Word Strategies ( 2 Read passages to build on skills and improve 
comprehension 

c. The Code ( 3 Practice in syllable strategies and word analysis 

d. Sight Words ( 4 Direct instruction and practice in letter-sound 
correspondences 

21. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 program. Mark (X) only one box per row. 
Strongly Strongly 
| understand... disagree agree 
a. how the System 44 software works to 0, O, 0, oO, Oo, 
individualize instruction and practice for students. 

b. what to do during small group instruction. 0, O, 0, Oo, Oo, 

c. how to use SAM to group students for small 0, me 0, Oo, Oo, 
group instruction. 

d. when to use the Decodable Digest, 44Book, and 0, Oo, 0, Oo, Oo, 
System 44 library books for student practice. 

e. howto monitor student progress in System 44. 0, 0, 0, Oo, Oo, 


RMC Research Corporation @Portland, OR System 44 Study Design 


19. Some questions | have about System 44 are... 
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System 44 Teacher Survey 


Ni [Uraals)ts ma Olallitsvemexe alele) mB) isia(er! 


April 2011 


About This Survey 


This survey contains questions about your background and questions about the 

System 44 program that you implemented this school year. Your responses are important 
in helping us understand which factors contribute to the success of the program and what 
issues need to be addressed. 


Please write your answers directly on the survey by checking the appropriate boxes or by 
writing your response in the space provided. Your individual responses will be kept 
confidential and will not be shared with your principal or other school or district personnel. 
Summary data from this survey will be shared with Scholastic staff to assist in 
understanding System 44 implementation. 


We expect it will take approximately 15 minutes to complete the survey. 


Thank you very much for your help. 


Prepared by: 
RMC Research Corporation 
111 SW Columbia Street 
Suite 1200 

RESEARCH Portland, OR 97201 


Professional Development 


1. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 training activities provided this year. Mark (X) only one box per row. 
Strongly Strongly 
disagree agree N/A 
a. The group training on System 44 prepared 0, O. 0, Oo, he Oo, 
me to use the program in my classroom. 
b. The individual support from Scholastic during 0, Oo, 0, OO, Oo, 0, 


the year enhanced my skills in using System 
44 in my classroom. 


c. The individual support from district staff iP o. 0, ly Oo, oO, 
during the year enhanced my skills in using 
System 44 in my classroom. 


d. lam pleased with the amount of System 44 0, o. 0, Oo, Oo, Oo, 
professional development | received. 
e. lam pleased with the quality of System 44 0, o. 0, Oo, Oo, Oo, 
professional development | received. 
2. Which training or support topics were most helpful to you as you implemented System 44? 
3. What additional System 44 training or support would have been helpful? 


Implementation of System 44 Components 


4. How frequently did you use each of the following System 44 components? 
Rarely or Oncea Every 
never week day 

a. Whole class instruction in System 44 0, O. Oo, O, OO, 
b. Teaching Guide (bound copy) Oo, O. Oo, OO, 1, 
c. Teaching Guide (computer version) 0, O., Oo, Oo, O, 
d. 44 Book or Decodable Digest (computer version) Oo, [Ele 0, O, OO, 
e. SAM reports a, OF OF &% OO; 
f. Small group instruction: SMART lessons Oo, Oo, 0, Oo, Oo, 
g. Individual student support (outside of small group) Oo, Oo, 0, Ed; OO, 
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5, How frequently did you supplement your System 44 instruction with the following materials? 


Rarely or Once a Every 
never week day 
a. SAM practice pages on f= OF &% OO; 


b. SAM book expert (identify appropriate books) ie cE ide “ele »-Gle 


c. Flip chart O, Oo OF © OO; 
d. Letter manipulatives O, Oo OF © OO, 
e. System 44 audiobooks on f% OF &% OO; 
f. Reading Counts! quizzes O, Oo OF © OO; 
g. Sound and Articulation DVD 0, Oo Of, &% OO, 
h. Conference Guides (for 44 Library books) 0, Fis: Ty. sy.. 
i. Scholastic Red routines 1, Bs se. Hy Gs 
j. Other materials that were not System 44 0, 1, O, 1, Oo, 
6. On average, how many minutes did each student use System 44 on the computer each day? 


How many computers were available in your classroom for System 44 instruction? 


Instructional Practices 


7. Please indicate which of the following teaching strategies and materials you used with your 
struggling readers this year (2010-2011). Mark (X) only one box per row. 


Central to Small part of Not part of 
my reading my reading my reading 


instruction instruction instruction 
Instruction 

a. Provide time in reading block for students to O, Oo, 0, 
practice skills on their own. 

b. Provide materials for at-home practice of skills 0, 1, Oo, 
introduced in class. 

c. Provide extra reading instructional time for 0, 1, O, 
struggling readers. 

d. Include writing opportunities in reading instruction. O, 0, 0, 

e. Build spelling practice into reading instruction. 1, Oo, 0, 

f. Develop reading skills using science and social O, Oo, 0 


studies texts. 
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Central to Small part of Not part of 
my reading my reading my reading 


instruction instruction instruction 
Grouping 
g. Teach whole class reading lessons. O, 0, 0, 
h. Work one-to-one with students on reading. 1, 1, 0, 
i. Work with small groups of students. O, Oo, 0, 
j. Group students based on skill levels. 0, 1, O; 
k. Group students based on need for additional 0, O. Oo, 
instruction in specific, targeted skills. 
|. Group students based on mixed abilities (pairs or 0, O. O, 
cooperative groups). 
Reading Materials 
m. Use core reading series. O, 0, 0, 
n. Use supplementary reading materials. OF, O me 
o. Use trade books. O, Oo, Oo, 
p. Use books that are easy to decode. 0, 1, ime 
q. Use separate intervention materials for some 0, O. OH, 
students. 
r. Use reading software/technology. O, 0, 0, 
s. Use teacher-made materials. 0, OO, mp 
Assessments 
t. Use test results to organize instructional groups. OF, Oo, 0, 
u. Use informal reading inventories. OF, Oo, 0, 
v. Conduct miscue analysis, analyzing errors 0, 1, OH, 
students make while reading aloud. 
w. Use tests to determine progress on skills. O, Oo, 0, 
x. Use diagnostic tests to identify students who need 0, 1. O, 


reading intervention services. 
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8. Please indicate which of the following reading instructional activities you used with your 
struggling readers this year (2010-2011). Mark (X) only one box per row. 


Central to Small part of Not part of 
my reading my reading my reading 


instruction instruction instruction 
Reading Text 
a. Students reread familiar text. O, Oo, 0, 
b. Students confirm or revise predictions after O, Oo, 0, 
reading. 
c. Students generate their own questions about text 0, O. Oo, 
material. 
d. Students identify their comprehension break-downs 0, Oo, 0, 
and use fix-up strategies with a partner. 
e. Students orally summarize main events in stories 0, O. O, 
and informational texts. 
f. Students use graphic and semantic organizers to 0, O. O, 
track information. 
Work With Sounds and Words 
g. lIteach specific strategies for decoding unfamiliar O, Oo, 0, 
words. 
h. |teach decoding/phonics skills while reading O, Oo, 0, 
stories. 
i. Students practice reading high frequency words for 0, O. O, 
automaticity. 
j. Students use knowledge of root words, prefixes, OF, Oo, 0, 
and suffixes to decode new words. 
k. Students work with prefixes and suffixes to change O, Oo, 0, 
the meaning of words. 
|. Students use context clues to identify unknown 0, O., OH, 
words. 
m. I discuss new and unusual words before reading. O, Oo, 0, 
Other Techniques 
n. Students answer questions in writing after reading 0, Oo, 0, 
stories. 
0. Students select books from the library for 0, 0, O, 
independent reading. 
p. Students are given time to read on their own for 0, Oo, I; 


enjoyment. 
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9. Please rate the extent to which you agree or disagree with the following statements. Mark (X) 
only one box per row. 


My System 44 students this year received enough 


instruction and practice in the following areas to Strongly Strongly 
make sufficient gains in reading. disagree agree 
a. phonemic awareness of Gs BO: Ge OD; 
b. phonics Oo OF Ds GG: 
c. fluency Ey s- Ee We “Ele 
d. vocabulary Gy ‘Ei <a jh: “Ge 
€. comprehension GG Be 6; (By Gs 
System 44 Concepts 


The following questions are intended to capture your current understanding of the System 44 program 
based on either experience with the program or what you learned in the training. This information will 
help us to understand areas that the training did or did not address, and will help guide future training 
sessions. 


10. Which of the following is NOT a skill that is measured by the Scholastic Phonics Inventory? 
Mark (X) only one box. 


1, Identify sight words 
CO, Recognize letter names 
Identify nonsense words 
1, Improve phonological awareness 
11. Which of the following System 44 program components is designed to provide students with 
written practice in applying the skills they are learning in the software? Mark (X) all that apply. 
O), Decodable Digest 
1, 44Book 
1. Flip chart 
Oz SAM Practice Pages 
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12. Which of the following System 44 program components is designed to provide reading fluency 
practice for students? Mark (X) all that apply. 


O, Decodable Digest 
1, System 44 Library Books 
1, System 44 Audiobooks 


1, Success Strand 


13. Scholastic Reading Counts! is ... Mark (X) only one box. 
Oj, the System 44 adaptive software that continuously collects data on student performance. 
©. aset of quizzes that measure comprehension of each System 44 paperback book. 
© aset of progress monitoring tools given at the end of each System 44 series. 


©, aset of motivational videos that students can play when they have successfully completed 
a System 44 series of lessons. 


14. Which of the following is NOT a feature of System 44? Mark (X) only one box. 
1, Using SAM reports for instructional grouping 
©. Teacher-led instruction in small groups 
Library books that reinforce specific phonics skills 


1, Successful completion of every software lesson 


15. Phonemic awareness is . . . Mark (X) all that apply. 
the foundation for phonics instruction. 
1, the ability to hear and manipulate individual sounds in spoken words. 
Oj. the same as phonological awareness. 
1, the recognition that sentences consist of separate words. 
16. The Scholastic Achievement Manager (SAM) can. . . Mark (X) all that apply. 
O. help teachers form instructional groups for students who need work on the same skills. 
O., provide detailed, diagnostic information about student strengths and weaknesses. 
allow the teacher to monitor students’ ongoing progress on the software. 


1, provide downloadable assessments for various reading skills. 
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17. Select the description from the right column that matches each of the four System 44 Zones 
listed on the left and enter the matched number in the box: 


a. Fluency Zone [__) 1 Explicit instruction in the 44 sounds of the English 
language and the letters that make each sound 


b. Spelling Zone [ 2 Reading of decodable text with a focus on comprehension 
c. Word Zone [ 3 Instruction and practice in specific spelling rules 
d. Smart Zone [_) 4 Modeling and practice in blending sounds 


18. Select the description from the right column that matches each of the four System 44 strands of 
instruction listed on the left and enter the matched number in the box: 


Success ( 1 Builds automatic recognition of high frequency words 
Word Strategies [ 2 Read passages to build on skills and improve 


comprehension 
The Code [ 


C. 
d. Sight Words ae 


oo 


Practice in syllable strategies and word analysis 


aN 


Direct instruction and practice in letter-sound 
correspondences 


19. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 program. Mark (X) only one box per row. 


Strongly Strongly 
| understand... disagree agree 
a. how the System 44 software works to 0, El 0, Oo, OO 
individualize instruction and practice for students. 

b. what to do during small group instruction. 0, O, 0, Oo, O 

c. howto use SAM to group students for small 0, 0, 0, Oo, O, 
group instruction. 

d. when to use the Decodable Digest, 44Book, and 0, Oo, 0, Ely Oo, 
System 44 library books for student practice. 

e. howto monitor student progress in System 44. 0, 0, 0, oO, ee 
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20. Some comments | have about System 44 are... 


RMC Research Corporation @Portland, OR System 44 Teacher Survey 


Teacher Survey 


Murrieta Valley Unified School District 


April 2011 


About This Survey 


This survey contains questions about your background and questions about your reading 
instruction this school year. Your responses are important in helping us learn more about 
reading instruction in your district. 


Please write your answers directly on the survey by checking the appropriate boxes or by 
writing your response in the space provided. Your individual responses will be kept 
confidential and will not be shared with your principal or other school or district personnel. 
Summary data from this survey will be shared with Scholastic staff as part of a System 44 
study that is taking place in your district. 


We expect it will take approximately 15 minutes to complete the survey. Please return the 
survey and accompanying consent form to us in the self-addressed stamped envelope 
we provided. Upon receipt of this survey, we will email you a $25 gift certificate to the 
Scholastic Teacher Store. 


Thank you very much for your help. 


Prepared by: 
RMC Research Corporation 
111 SW Columbia Street 
Suite 1200 

RESEARCH Portland, OR 97201 


Teacher Background 


1. Are you female or male? 
O, Female 
Oo, Male 
2. How many years have you worked as a full-time teacher in public schools ? Write in the number 


of years below. Count part of a year as one year. 

Number of years 
a. Total number of years as a teacher (do not include student teaching) 
b. Total number of years as a teacher in Murrieta Unified School District 


c. Total number of years as a teacher at your current school 


3. What is the highest degree you have obtained as of September 2010? Mark (X) only one box. 
©, Bachelor's (B.A., B.S., B.E., etc.) 
1, Master’s degree (M.A., M.A.T., M.B.A., M.Ed., M.S., etc.) 
O, Education specialist or certification at least one year beyond master’s level 


(1, Doctorate or professional degree (Ph.D., Ed.D., M.D., L.L.B., J.D., D.D.S.) 


O.; Other (please specify: ) 
4. Which of the following describes the teaching certificate(s) you currently hold in California? 
Mark (X) all that apply. 
1, Single subject O, Reading Specialist 
, Multiple subject Oj, Administrative Services 
O, Preliminary 1, District Intern 
4 Professional Clear © CLAD or BCLAD (Bilingual, Crosscultural, 


Language and Academic Development 
O. Education Specialist Instruction ere P ) 


5. | have used System 44 in my classroom. 
1, Yes Ol, No 

6. | have used Read 7180 in my classroom. 
O, Yes O1, No 
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Professional Development 


re Have you had training or support in System 44 (e.g., in-classroom coaching, group training) 
provided by a Scholastic representative or other experienced System 44? Mark (X) all that 
apply. 
O., Have not received any training in System 44 
O1, Attended a System 44 training 
1, Received other support for System 44 from a Scholastic representative 
1, Received other support from another experienced System 44 user 
O. Other (please specify: ) 


Instructional Practices 


8. Please indicate which of the following teaching strategies and materials you used with your 
struggling readers this year (2010-2011). Mark (X) only one box per row. 


Central to Small part of Not part of 
my reading my reading my reading 


instruction instruction instruction 
Instruction 

a. Provide time in reading block for students to O, Oo, 0, 
practice skills on their own. 

b. Provide materials for at-home practice of skills 0, 1. O, 
introduced in class. 

c. Provide extra reading instructional time for O, Oo, 0, 
struggling readers. 

d. Include writing opportunities in reading instruction. O, Oo, 0, 

e. Build spelling practice into reading instruction. 1, Oo, 0, 

f. Develop reading skills using science and social O, Oo, 0, 
studies texts. 

Grouping 

g. Teach whole class reading lessons. OF, Oo, me 

h. Work one-to-one with students on reading. 1, O., O, 

i. Work with small groups of students. O, Oo, 0, 

j. Group students based on skill levels. OF, 1, Oj; 

k. Group students based on need for additional 0, O. Oo, 
instruction in specific, targeted skills. 

|. Group students based on mixed abilities (pairs or O, Ey 0, 


cooperative groups). 
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Reading Materials 


m. Use core reading series. 

n. Use supplementary reading materials. 

o. Use trade books. 

p. Use books that are easy to decode. 

q. Use separate intervention materials for some 
students. 

r. Use reading software/technology. 

s. Use teacher-made materials. 

Assessments 

t. Use test results to organize instructional groups. 

u. Use informal reading inventories. 

v. Conduct miscue analysis, analyzing errors 
students make while reading aloud. 

w. Use tests to determine progress on skills. 

x. Use diagnostic tests to identify students who need 


reading intervention services. 
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Ceniral to 
my reading 
instruction 


0, 
OH, 
0, 
0, 
0, 


ao, 
a, 


a, 
a, 
a, 


a, 
a, 


Small part of 
my reading 
instruction 


0, 
0. 
0, 
0. 
0. 


Ds: 
D2 


Ds: 
0: 
D2 


Ds: 
D2 


Not part of 
my reading 
instruction 


Os 
Os 
Ds 
Os 
0; 


Ds 
Os; 


Ds 
Ds 
Os 


OD: 
Os; 


9. Please indicate which of the following reading instructional activities you used with your 
struggling readers this year (2010-2011). Mark (X) only one box per row. 


Central to Small part of Not part of 
my reading my reading my reading 


instruction instruction instruction 
Reading Text 
a. Students reread familiar text. 1, O. 0, 
b. Students confirm or revise predictions after 0, O., OH, 
reading. 
c. Students generate their own questions about text O, Oo, 5 
material. 
d. Students identify their comprehension break-downs 0, O. O, 
and use fix-up strategies with a partner. 
e. Students orally summarize main events in stories 0, Oo, 0, 
and informational texts. 
f. Students use graphic and semantic organizers to O, Oo, Eas 
track information. 
Work With Sounds and Words 
g. lIteach specific strategies for decoding unfamiliar 0, O., O, 
words. 
h. [teach decoding/phonics skills while reading 0, O., Oo, 
stories. 
i. Students practice reading high frequency words for 0, Oo, 0, 
automaticity. 
j. Students use knowledge of root words, prefixes, 0, O. O, 
and suffixes to decode new words. 
k. Students work with prefixes and suffixes to change 0, O. O, 
the meaning of words. 
|. Students use context clues to identify unknown O, Oo, Cy; 
words. 
m. | discuss new and unusual words before reading. 1, O., OH, 
Other Techniques 
n. Students answer questions in writing after reading 0, Oo, a 
stories. 
o. Students select books from the library for 0, Oo, 0, 
independent reading. 
p. Students are given time to read on their own for 0, O. O, 


enjoyment. 
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10. What reading instructional program(s) did you use this year with your struggling readers? 


Mark (X) all that apply. 
O,, Houghton-Mifflin O, REACH 
O» Morphographs On Explode the Code 
O. Ticket to Read O Rosetta Stone 
O4 Read Naturally O| Language! 
O SRA Decoding OO, Other 
O; Voyager 07, Other 
11. Please rate the extent to which you agree or disagree with the following statements. Mark (X) 


only one box per row. 


My struggling students this year have received 


enough instruction and practice in the following Strongly Strongly 
areas to make sufficient gains in reading. disagree agree 
a. phonemic awareness Oo Oo GOs se 
b. phonics a, oF OF % OO; 
c. fluency Ee “Ge “Gy. ay soe 
d. vocabulary Oo Os Ge “Bh Gs 
e. comprehension Ay Gy. Oy (Be Ee. 
Thank you! 
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System 44 


Murrieta Unified School District 
Classroom Visit 


School: 0, Alta Murrieta O,; Monte Vista O, Shivela 
O, Avaxat O, Murrieta O, Thompson 
OH, Buchanan 6, Rail Ranch O10 McElhinney 
O, Curran O,, Warm Springs 
ry i aphaeee Corporation 


111 SW Columbia Street 
Suite 1200 
RESEARCH Portland, OR 97201 
Fall Winter Spring 
2010 2011 2011 


Date: Class Visit #: 1 2 3 
Number of Students in Class During Visit: Teacher Name: 
Observer: System 44 start time: System 44 end time: Total minutes: 


Grade Levels Represented: O, 4° oF, 5" Gg, 6" og, 7" of 8" 
Class Type: Oj, Special Day Class O,, Pullout Class 0; Replacement ELA Class (mark aii that apply.) 
Number of Computers Available for System 44: 


Number of Other Adults in Class: Role/Activity of Other Adult(s): 


Teaching or organizational strategies used: 


Notes: 


Interval start time: Interval end time: Total minutes: System 44 Classroom Visit Protocol 


01 Check here if no whole class activity takes place on date of observation OV Check here if whole class activity takes place at end of class 
Teacher-Led Activity Teacher-Led Support 
What is the teacher doing, and how fully is the What supports is the teacher using, 
teacher implementing each activity? and how fully” is the teacher using them? 
0 
O,, Teaching Guide’ O, Letter manipulatives @O®@ 
3) e Introducing @DO®@ O.. Flip chart @O® 
62 e Teaching/modeling OO®@ Oy 44 Book ODO®@ 
ae e Engaging students in guided practice @O®@ O, Decodable Digest @Ode@ 
g 2 e Apply and reinforcing/reviewing @O@O® O; Teaching Resources for the System 44 Library @DO®@ 
Q Notes On, System 44 paperback books @©O®@ 
- : The “I Do, We Do, You Do” components should be incorporated for full On SAM Resources/Worksheets OO®@ 
@ ares . . os Oj} Sound and articulation DVD OO@ 
2 ~ | iesonsmentuine sudenoncard povangvobalerwnicneodasfa. [CH SAM Data Reports (List . __ 000 
needa a earn a material but does not provide any guidance, O. Non-System Woaaieriale ODO 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


Independent (Individual/Paired) Activity Independent (Individual/Paired) Student Support 
What are individual students or pairs What supports are the students using, 

doing during large group instruction? Mark all that apply. and how engaged’ are students in their use of them? 
O,,. Writing activity using System 44 materials O, System 44 audiobooks # Ss @DO®@ 
» ® |» Reading activity using System 44 materials Oh 44 Book # Ss @O®@ 
i 3 O. Other practice activity using System 44 materials O. Decodable Digest # Ss @O®@ 
E 5 On SA Le SSSR oo O4 SAM Resources/Worksheets #Ss_ |: ODO 
2 O. Non-System 44 instructional activity O. System 44 paperback books (without audio) | #Ss @DO®@ 
< — |0; Non-instructional activity O; Letter manipulatives #Ss OO®@ 
O, Sound and articulation DVD #Ss @O®@ 


3 Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


Interval start time: 


Interval end time: 


Total minutes: 


System 44 Classroom Visit Protocol 


Teacher-Led Activity 


What is the teacher doing, and how fully is the teacher 


implementing each activity? 


Teacher-Led Support 


What supports is the teacher using, 


and how fully” is the teacher using them? 


-_~ 

” 

c 
5 9 
Oo 5 
O® 
7] 
= 
(Jp) 


( 


(# minutes: ) 


O, Teaching Guide’ O1, Decodable Digest OO®@ 
e Introducing @@O@ |0. 44 Book @D®@ 
e Teaching/modeling @@@ |Oq Teaching Resources for the System 44 Library @D®@ 
e Engaging students in guided practice | ©O@ | HO, Letter manipulatives @©O@ 
e Apply and reinforcing/reviewing @@O®@ | 0; Flip chart @O@D®@ 

Notes. O7,, Sound and articulation DVD ODO® 

} The “I Do, We Do, You Do” components should be incorporated for full 1, SAM Resources/Worksheets OD®@ 

implementation. i 

? Full use of supports involves modeling answers and guiding student Oj System ae paperback books @Q@O®@ 

usage of lessons; monitoring student work; and providing verbal or written Oj, SAM Data: Reading Progress Report @O® 

feedback. If a teacher assigns an activity using a material but does not ! = = : = 

provide any guidance, monitoring, or feedback, mark 0. O;1 SAM Data: Differentiated Instruction Grouping Report @O®@ 

Om Non-System 44 materials O71 SAM Data: Response to Intervention Report @O®@ 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


System 44 computer software 


How engaged are students in their use of the software? 


System 44 computer software ( 


students) 


© Not at all engaged/none engaged 


®O Partially engaged/some engaged 


@ Fully engaged/all engaged 


Independent (Individual/Paired) Activity 


What are individual students or pairs 
doing during large group instruction? Mark all that apply. 


Independent (Individual/Paired) Student Support 
What supports are the students using, 


and how engaged’ are students in their use of them? 


Individual/Pairs 


students) 


( 


O. Writing activity using System 44 materials O, System 44 audiobooks #Ss @DO®@ 
O») Reading activity using System 44 materials OL 44 Book # Ss @O®@ 
O, Other practice activity using System 44 materials O., Decodable Digest # Ss @O®@ 
O14 Self monitoring using progress check document, O, SAM Resources/Worksheets #Ss OM® 
reading logs, Reading Counts quizzes, etc. = 

O Non-System 44 instructional activity O. System 44 paperback books (without audio) | # Ss @O®@ 
O, Non-instructional activity OF, Letter manipulatives # Ss ODO®@ 

O, Sound and articulation DVD #Ss_ | OOO 


5 Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


Interval start time: 


Interval end time: 


Total minutes: 


System 44 Classroom Visit Protocol 


Teacher-Led Activity 


What is the teacher doing, and how fully is the teacher 


implementing each activity? 


Teacher-Led Support 


What supports is the teacher using, 


and how fully” is the teacher using them? 


-_~ 

” 

c 
5 9 
Oo 5 
O® 
7] 
‘= 
wo 


( 


(# minutes: ) 


O, Teaching Guide’ O1, Decodable Digest OO®@ 
e Introducing @@O@ |0. 44 Book @D®@ 
e Teaching/modeling @@@ |Oq Teaching Resources for the System 44 Library @D®@ 
e Engaging students in guided practice | ©O@ | HO, Letter manipulatives @©O@ 
e Apply and reinforcing/reviewing @@O®@ | 0; Flip chart @O@D®@ 

Notes. O7,, Sound and articulation DVD ODO® 

} The “I Do, We Do, You Do” components should be incorporated for full 1, SAM Resources/Worksheets OD®@ 

implementation. i 

? Full use of supports involves modeling answers and guiding student Oj System ae paperback books @Q@O®@ 

usage of lessons; monitoring student work; and providing verbal or written Oj, SAM Data: Reading Progress Report @O® 

feedback. If a teacher assigns an activity using a material but does not ! = = : = 

provide any guidance, monitoring, or feedback, mark 0. O;1 SAM Data: Differentiated Instruction Grouping Report @O®@ 

Om Non-System 44 materials O71 SAM Data: Response to Intervention Report @O®@ 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


System 44 computer software 


How engaged are students in their use of the software? 


System 44 computer software ( 


students) 


© Not at all engaged/none engaged 


®O Partially engaged/some engaged 


@ Fully engaged/all engaged 


Independent (Individual/Paired) Activity 


What are individual students or pairs 
doing during large group instruction? Mark all that apply. 


Independent (Individual/Paired) Student Support 
What supports are the students using, 


and how engaged’ are students in their use of them? 


Individual/Pairs 


students) 


( 


O. Writing activity using System 44 materials O, System 44 audiobooks #Ss @DO®@ 
O») Reading activity using System 44 materials OL 44 Book # Ss @O®@ 
O, Other practice activity using System 44 materials O., Decodable Digest # Ss @O®@ 
O14 Self monitoring using progress check document, O, SAM Resources/Worksheets #Ss OM® 
reading logs, Reading Counts quizzes, etc. = 

O Non-System 44 instructional activity O. System 44 paperback books (without audio) | # Ss @O®@ 
O, Non-instructional activity OF, Letter manipulatives # Ss ODO®@ 

O, Sound and articulation DVD #Ss_ |: OOO 


5 Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


Interval start time: Interval end time: 


Total minutes: 


System 44 Classroom Visit Protocol 


Teacher-Led Activity 


implementing each activity? 


What is the teacher doing, and how fully is the teacher 


Teacher-Led Support 


What supports is the teacher using, 


and how fully” is the teacher using them? 


O. Teaching Guide’ O1, Decodable Digest O©DO@ 

e Introducing @O®@ |0. 44 Book @D®@ 

oy e Teaching/modeling @@@ | Oa Teaching Resources for the System 44 Library @D®@ 

= 5 | e Engaging students in guided practice | O©O@ | O. Letter manipulatives @D®@ 

os e Apply and reinforcing/reviewing @@O®@ | 0; Flip chart @O® 

OG | Notes. O, Sound and articulation DVD @O® 

® Ss ' The “I Do, We Do, You Do” components should be incorporated for full 1, SAM Resources/Worksheets OD® 

& o£ implementation. iD 

iS 2 Full use of supports involves modeling answers and guiding student Oj System 44 paperback books @Q@O®@ 

~~ | usage of lessons; monitoring student work; and providing verbal or written | [, SAM Data: Reading Progress Report O@O® 
feedback. If a teacher assigns an activity using a material but does not : = = = = 

provide any guidance, monitoring, or feedback, mark 0. O1 SAM Data: Differentiated Instruction Grouping Report @O®@ 

Om Non-System 44 materials ©) SAM Data: Response to Intervention Report @O®@ 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


System 44 computer software 


How engaged are students in their use of the software? 


System 44 computer software ( students) 


© Not at all engaged/none engaged 


® Partially engaged/some engaged 


@ Fully engaged/all engaged 


What are individual students or pairs 
doing during large group instruction? Mark all that apply. 


Independent (Individual/Paired) Activity 


Independent (Individual/Paired) Student Support 


What supports are the students using, 


and how engaged’ are students in their use of them? 


students) 
go 


Individual/Pairs 


( 


O. Writing activity using System 44 materials O, System 44 audiobooks #Ss @DO®@ 

Oh» Reading activity using System 44 materials O» 44 Book # Ss @O®@ 

Other practice activity using System 44 materials O1, Decodable Digest # Ss @O®@ 

O, Self monitoring using progress check document, O, SAM Resources/Worksheets #Ss OdO® 
reading logs, Reading Counts quizzes, etc. Saas 

O.. Non-System 44 instructional activity Ol System 44 paperback books (without audio) | # Ss @O®@ 

O, Non-instructional activity OF Letter manipulatives # Ss ODO®@ 

O, Sound and articulation DVD #Ss_ |: ODO 


5 Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


System 44 Classroom Visit Protocol 


Overall 
Did the teacher: Notes 

[1] Maintain an appropriate pace? @©DO® 
[2] Deliver lessons appropriate for the skill levels of the students? @©DO®@ 
[3] Appear well prepared for the lesson? @©DO® 
[4] Actively tries to engage students throughout the lesson? @©DO® 
[5] Maintain a positive learning environment (e.g., limited interruption, OOO® 
good classroom management)? 

[6] Execute the lesson well? ODO®@ 


[7] Monitor students? (Check for on-task behavior, understanding; provide 
feedback) OVO® 


OVERVIEW 


[8] Keep students on task? ODO®@ 


[9] Ensure smooth and efficient transitions between rotations (e.g., use of 
timer or clock, evidence of transition routines, rituals for students putting @©DO® 
materials away). 


[10] Use flexible groupings for students (using SAM data, anecdotal ODO® 
evidence) or regroup students based on instructional needs? 


[11 Differentiate support based on students’ needs (e.g., provided O©DOO@® 
individualized instruction, used ELL or SPED sections of teaching guide) 


4-point scale (0 = Not at all, 1 = To a small extent, 2 = To a moderate extent, 3 = Definitely) 


System 44 Classroom Setup Notes 
Computers (for 1/3 of class) are accessible and functioning @O©D® 
wn) 
”) a Auxiliary equipment (headsets, microphones, CD players) are accessible 
< Ne @O®@ 
_9O | and functioning 
oa 
System 44 materials are easily accessible to students and teacher @O®@ 


3-point scale (0 = None, 1= Some, 2 = All) 


School: 


System 44 
Teacher Interview 
Spring 2011 


Thank you for taking the time to talk with me today. As you know, RMC Research is 
evaluating the implementation of the System 44 program in your school district this 
year. We are talking with teachers who implemented the program to help us understand 
factors that may have contributed to or hindered the success of the program. Your 
responses will be kept confidential and will not be shared with your principal or other 
district personnel. Your answers will be combined with other teachers’ information so 
that no one can know about any one person’s responses. 


Do you have any questions before we begin? 


1. Had you taught System 44 before this school year? 


O yes Ono 


If yes, how many years have you used the program? 


2. What features of the System 44 program do you like best? 


3. What aspects of System 44 implementation do you think you struggled with the 
most this year? How did you address or resolve the challenge(s)? 


4. How did you decide how to group students on any given day (e.g., which 
students would use the computer first)? 
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How frequently did you use whole group instruction? (If once a week or more: 
what did you typically focus on during whole group instruction?) (If less than once 
a week: why did you not use whole group instruction more frequently?) 


In what ways, if any, did you use SAM reports? Which reports did you use, and 
how frequently did you use them? Prompt: (Show report examples) 


a. Screening and Placement; 

b. Software Performance; 

c. Reading Progress; 

d. Response to Intervention; 

e. Differentiated Instruction Grouping; 


f. Student Mastery 


What technology issues did you experience beyond the start-up period? Did 
these issues get in the way of implementing the program? 


Did you experience any issues with scheduling students to participate in 
System 44? If so, please explain whether these issues resolved during the 
school year. 


If you had the opportunity to make recommendations to teachers in another 
school that is planning to implement System 44 for the first time next year, what 
advice would you give them? 


System 44 Teacher Interview 


School: 


System 44 
Principal Interview 
Spring 2011 


As part of the evaluation of the Scholastic System 44 program, we want to talk with 
administrators at each participating school to learn more about the context in which System 44 
was implemented as well as factors that facilitated or hindered implementation of the program. 


1. 


In what year did your school first implement the System 44 reading program? 


0 2010-2011 
Oo 


In what year did your school first implement the Scholastic Read 180 reading program? 


0 2010-2011 
0 


Have you had an opportunity to observe System 44 classes at your school? If yes, what 
features of the program do you think contribute to student success? 


What other programs or services are available to students in your school who are 
struggling with reading? (Prompts: in-class programs, pull-out programs, after school 
programs or tutoring) 


What factors do you think have facilitated implementation of System 44? (Prompts: 
district staff support, Scholastic staff support, teacher interest, student interest, 
computer-delivered instruction, teacher-delivered instruction, books and other materials) 
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10. 


11. 


What factors have hindered implementation of System 44? (Prompts: hardware issues, 
software issues, problems with other materials such as books, testing to identify the right 
students, testing for treatment and control group students, pullout from other subjects, 
research requirement for random assignment, problems at Thompson and Curran) 


What feedback have you received from teachers or parents about student outcomes 
resulting from participation in System 44?) 


What feedback have you received from teachers about the training or support they 
received from Scholastic for implementing the program? 


What plans does your school have for continuing System 44 next year? (Prompts: Do 
you plan to do anything differently as far as implementation or scheduling?) 


Do you have any suggestions for how the evaluation might have been better 
implemented? (e.g., identifying students, random assignment, SPI and SRI testing, 
individual testing of treatment and control group students). 


Are there any other comments you would like to make? 


System 44 Principal Interview 


System 44 
District Staff Focus Group 
Spring 2011 


As part of the evaluation of the Scholastic System 44 program, we want to talk with 
district staff involved in implementing the program to learn more about the context in 
which System 44 was implemented as well as factors that facilitated or hindered 
implementation of the program. 


System 44 Implementation 


1. What types of professional development did System 44 teachers receive this year? 


What did you or the System 44 teachers find most useful about the professional 
development they received? 


What suggestions, if any, do you have regarding professional development? 


2. What factors do you think facilitated implementation of System 44 in the 11 study 
schools this year? 


3. What factors do you think hindered implementation (i.e., what challenges did schools 
face in implementing System 44 in their classrooms)? 
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4. What recommendations would you have for another district that is starting to use the 
System 44 program in their schools (i.e., what were the lessons learned)? 


5. Overall, what impact do you think the program has had on students? What aspects of 
the program do you believe have had the greatest impact on students? 


Evaluation Implementation 


6. What questions would you be interested in having answered by the evaluation of System 
44 in your district? 


7. What recommendations would you make concerning the evaluation component of the 
System 44 study (i.e., what were the lessons learned)? 


8. What was the process for administering the SPI and SRI tests in Fall and Winter (i.e., 
who was responsible for administering them, how long did the process take)? 


9. Is there anything else that you would like to add? 
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Appendix B 
Description of the Counterfactual 


Exhibit B1 
Summary of Reading and Language Arts Interventions in 
Control Group Elementary Schools 


# of Control # of Control 
Group Control Group 
School Students in Classroom Students in Reading Interventions Used 
Name School Number Class Class Structure in Class 
Alta 4 1 4 Small group SRA Decoding/spelling 
Murrieta direct instruction through Morphographs 
Avaxat 14 2 10 Pullout Voyager-Passport 
3 2 Core class High Group/HM 
4 2 Core class Middle Group/HM 
Buchanan 16 5 2 Self-contained Ticket to Read, Read Naturally 
6 3 Self-contained Ticket to Read, Read Naturally 
7 3 Self-contained Ticket to Read, Read Naturally 
8 1 Self-contained Ticket to Read, Read Naturally 
9 2 Self-contained Ticket to Read, Read Naturally 
10 1 Team teaching Ticket to Read, Read Naturally 
11 2 Team teaching Ticket to Read, Read Naturally 
12 2 Team teaching Ticket to Read, Read Naturally 
Curran 7 13 2 Pullout Voyager-Passport 
14 2 Special Day Class SRA Decoded Reading 
15 3 Pullout Voyager-Passport 
Monte 30 16 5 Voyager-Passport 
Vista 17 6 Voyager-Passport 
18 4 Voyager-Passport 
19 5 Voyager-Passport 
20 2 Voyager-Passport 
21 3 Voyager-Passport 
22 2 Voyager-Passport 
23 2 Voyager-Passport 
Murrieta 14 24 3 
25 8 
26 1 
27 1 
28 1 
29 ALL 
Rail Ranch 16 30 16 Pullout REACH 


Note. HM = Houghton-Mifflin. 
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School 


Shivela 


Thompson 


McElhinney 


Warm 
Springs 


Exhibit B2 


Summary of Reading and Language Arts Interventions in 
Control Group Middle Schools 


# of Control 
Group 
Students in 
School 


28 


20 


10 


17 


Control 
Classroom 
Number 


31 
32 


33 
34 
35 
36 
37 
38 
39 


40 
i 
42 
43 
44 
45 
46 
47 
48 
49 
50 
51 


52 


53 


54 
55 


# of Control 
Group 
Students in 
Class 


1 
5 


Class Structure 
Core replacement 


Core replacement 


SDC 
General LA 
Core replacement 
General LA 
SDC LA 
General LA 
STRATEGIC LA 


General LA 
General LA 
General LA 
General LA 
General LA 
General LA 
General LA 
General LA 
General LA 
SDC 
SDC 


RSP-extra class 


Replacement 
class 


SDC 


SDC behavior 
General LA 


Reading Interventions 
Used in Class 


Voyager-Journeys 


Voyager-Passport (all SDC 
students) 


Voyager-Journeys 
Core-HM 
Read 180 (opted out) 
Core-HM 
Voyager-Journeys (modified) 
Core-HM 


Support class with modified 
curriculum and small class 
size 


Core-HM 
Core-HM 
Core-HM 
Core-HM 
Core-HM 
Core-HM 
Core-HM 
Core-HM 
Core-HM 
Voyager-Journeys 
Voyager-Journeys 


Explode the Code, 
Language! 


Rosetta Stone, Explode the 
Code 


Voyager-Journeys, 
Language! 


Worksheets (various) 
Core-HM 


Note. SDC = Special Day Class, GEN LA = General Language Arts, HM = Houghton-Mifflin, RSP = Resource 
Specialist Program. 
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Appendix C 
Item Results for Instructional Practices 


Percentage of Teachers Using Various Strategies With Struggling Readers 


Exhibit C1 


System 44 System 44 Control Teachers 
Last Year (2009-2010) This Year (2010-2011) This Year (2010-2011) | 
: Not Small Not Small Not Small 

Instructional Strategy Part Part Central Part Part Central Part Part Central 

Instruction 

1. Provide time in reading block for 0 8 92 0 33 67 7 31 62 
students to practice skills on their own. 

2. Provide materials for at-home practice 17 33 50 42 33 25 16 31 53 
of skills introduced in class. 

3. Provide extra reading instructional time 17 0 83 0 25 75 9 33 58 
for struggling readers. 

4. Include writing opportunities in reading 0 17 83 0 25 75 0 20 80 
instruction. 

5. Build spelling practice into reading 0 67 33 8 58 33 13 36 51 
instruction. 

6. Develop reading skills using science 25 25 50 33 33 33 16 38 47 
and social studies texts. 

Grouping 

7. Teach whole class reading lessons. 0 42 58 0 67 33 0 31 69 

8. Work one-to-one with students on 0 58 42 0 25 75 16 42 42 
reading. 

9. Work with small groups of students. 0 0 100 0 17 83 13 27 60 

10. Group students based on skill levels. 0 0 100 0 8 92 11 20 69 

11. Group students based on need for 0 25 75 0 17 83 11 29 60 
additional instruction in specific, 
targeted skills. 

12. Group students based on mixed abilities 25 42 33 9 45 45 9 43 48 
(pairs or cooperative groups). 

Reading Materials 

13. Use core reading series. 8 17 75 25 17 58 9 11 80 

14. Use supplementary reading materials. 0 42 58 27 45 27 11 50 39 

15. Use trade books. 8 33 58 27 a7 45 44 33 23 

16. Use books that are easy to decode. 0 25 75 8 50 42 16 56 29 

17. Use separate intervention materials for 17 17 67 42 17 42 30 34 36 
some students. 

18. Use reading software/technology. 8 92 0 8 92 13 38 49 

19. Use teacher-made materials. 0 58 42 17 58 25 9 51 40 

Assessments 

20. Use test results to organize instructional 0 17 83 0 25 75 11 24 64. 
groups. 

21. Use informal reading inventories. 42 50 0 42 58 9 42 49 

22. Conduct miscue analysis, analyzing 42 50 25 17 58 18 40 42 
errors students make while reading 
aloud. 

23. Use tests to determine progress on 0 8 92 8 33 58 4 27 69 
skills. 

24. Use diagnostic tests to identify students 0 0 100 0 8 92 11 22 67 
who need reading intervention services. 
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Exhibit C2 
Percentage of Teachers Using Various Activities With Struggling Readers 


System 44 System 44 Control Teachers 
Last Year (2009-10) This Year (2010-11) This Year (2010-11) 
. aus Not Small Not Small Not Small 
Instructional Activity Part Part Central Part Part Central Part Part Central 
Reading Text 
1. Students reread familiar text. 0 8 92 8 17 75 4 18 78 
2. Students confirm or revise predictions 0 25 75 8 33 58 2 24 73 
after reading. 

3. Students generate their own questions 0 42 58 8 50 42 7 47 47 
about text material. 

4. Students identify their comprehension 25 50 25 33 42 25 24 47 29 
breakdowns and use fix-up strategies 
with a partner. 

5. Students orally summarize main events 8 33 58 8 17 75 4 13 82 
in stories and informational texts. 

6. Students use graphic and semantic 0 17 83 0 42 58 0 36 64 


organizers to track information. 


Work With Sounds and Words 


7. |teach specific strategies for decoding 0 33 67 0 33 67 11 31 58 
unfamiliar words. 


8. | teach decoding/phonics skills while 0 25 75 0 25 75 13 27 60 
reading stories. 

9. Students practice reading high 8 50 42 8 50 42 24 42 33 
frequency words for automaticity. 

10. Students use knowledge of root words, 0 42 58 0 42 58 2 7 71 
prefixes, and suffixes to decode new 
words. 

11. Students work with prefixes and suffixes 0 42 58 0 42 58 2 36 62 
to change the meaning of words. 

12. Students use context clues to identify 0 8 92 0 8 92 0 11 89 
unknown words. 

13. | discuss new and unusual words before 0 8 92 0 8 92 0 11 89 
reading. 


Other Techniques 


14. Students answer questions in writing 0 25 75 0 8 92 0 13 87 
after reading stories. 


15. Students select books from the library 8 17 75 0 17 33 4 20 76 
for independent reading. 


16. Students are given time to read on their 0 33 67 8 33 58 9 36 56 
own for enjoyment. 
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Appendix D 
Fixed and Random Effects and Intraclass Correlations 


Exhibit D1 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring TOSREC Scores 


Level Effect Impact (B) SE DF t p 

School Intercept 31.97 0.96 10 33.38 .000 

Student System 44 -1.88 1.19 333 -1.58 115 
Fall TOSREC Score 0.67 0.04 333 16.08 .000 
Sex 3.14 1.23 333 2.55 011 
Caucasian 1.31 1.30 333 1.00 317 
Special Education -2.93 1.50 333 -1.94 .052 
Elementary school level 4.49 2.02 333 2.22 .027 

Random Effects 

Level Variance Components Variance 

School Level 2 5.64 

Student Level 1 119.58 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 53.49 .190 

Student Level 1 227.62 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Level Effect 

School Intercept 

Student System 44 
Fall CTOPP Elision Score 
Sex 
Caucasian 


Special Education 


Elementary school level 


Exhibit D2 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring CTOPP Elision Scores 


Fixed Effects 


Impact (B) 
23.63 


0.54 
0.58 
-0.66 
0.46 
-1.33 
1.10 


SE 
0.57 
0.49 
0.04 
0.51 
0.54 
0.62 
1.20 


DF 
10 
337 
337 
337 
337 
337 
337 


t p 
41.12 000 
1.09 .276 
14.37 000 
-1.29 198 
0.84 401 
-2.15 .032 
0.92 360 


Level Variance Components Variance 
School Level 2 2.78 
Student Level 1 20.79 


Random Effects 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance Icc 
School Level 2 2.72 .069 
Student Level 1 36.62 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D3 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring Woodcock-Johnson Ill Word Identification Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t 

School Intercept 49.34 0.90 10 54.55 

Student System 44 0.65 0.84 331 0.77 
Fall Woodcock-Johnson III 0.90 0.03 331 27.08 
Word Identification Score 
Sex -0.90 0.86 331 -1.04 
Caucasian 0.39 0.93 331 0.42 
Special Education -2.33 1.11 331 -2.10 
Elementary school level 0.80 1.93 331 0.42 

Random Effects 

Level Variance Components Variance 

School Level 2 6.62 

Student Level 1 59.34 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance Icc 
School Level 2 70.99 .237 
Student Level 1 228.25 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D4 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring Woodcock-Johnson Ill Word Attack Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 48.74 0.89 10 54.69 000 

Student System 44 1.08 0.90 334 1.19 .234 
Fall Woodcock-Johnson Ill 0.77 0.03 334 22.27 000 
Word Attack Score 
Sex -3.24 0.93 334 -3.50 001 
Caucasian -0.52 0.99 334 -0.52 602 
Special Education -2.81 1.14 334 -2.47 014 
Elementary school level -0.07 1.91 334 -0.04 970 

Random Effects 

Level Variance Components Variance 

School Level 2 5.99 

Student Level 1 68.94 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 42.94 .180 

Student Level 1 195.18 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D5 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring TOWRE Sight Word Efficiency Scores 


Fixed Effects 


Level Effect Impact (B) 

School Intercept 66.98 

Student System 44 0.10 
Fall TOWRE Sight Word 0.89 
Efficiency Score 
Sex 0.19 
Caucasian -0.85 
Special Education -0.51 
Elementary school level 0.11 


SE 

0.44 
0.54 
0.03 


0.55 
0.58 
0.70 
0.94 


DF 


10 
337 
337 


337 
337 
337 
337 


Random Effects 


Level Variance Components Variance 
School Level 2 1.25 
Student Level 1 24.55 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance Icc 
School Level 2 3.51 0.31 
Student Level 1 109.28 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D6 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring TOWRE Phonetic Decoding Efficiency Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 33.65 0.48 10 70.69 000 

Student System 44 -0.19 0.50 337 -0.37 712 
Fall TOWRE Phonetic 0.88 0.03 337 35.12 000 
Decoding Efficiency Score 
Sex -0.99 0.52 337 -1.91 .057 
Caucasian -0.25 0.55 337 -0.45 652 
Special Education -0.30 0.64 337 -0.47 639 
Elementary school level -1.29 1.00 337 -1.29 197 

Random Effects 

Level Variance Components Variance 

School Level 2 1.66 

Student Level 1 21.57 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 1.85 .016 

Student Level 1 112.09 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D7 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring CST Scores 


Level Effect Impact (B) SE DF t p 

School Intercept 333.09 3.20 10 104.19 .000 

Student System 44 4.01 4.23 280 0.95 344 
Fall CST Score 0.45 0.07 280 6.76 .000 
Sex 7.70 4.35 280 1.77 .077 
Caucasian 7.21 4.61 280 1.56 119 
Special Education 15.10 5.28 280 2.86 .005 
Elementary school level 27.82 6.86 280 4.06 .000 

Random Effects 

Level Variance Components Variance 

School Level 2 57.76 

Student Level 1 1,269.46 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 363.55 .194 

Student Level 1 1,508.04 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D8 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SRI Scores 


Level Effect Impact (f) SE DF t p 

School Intercept 563.58 12.43 10 45.35 000 

Student System 44 45.02 16.64 325 2.71 .008 
Fall SRI Score 0.74 0.05 325 16.09 000 
Sex 1.21 16.96 325 0.07 944 
Caucasian 13.48 18.02 325 0.75 455 
Special Education -81.01 20.14 325 -4.02 000 
Elementary school level -4.17 26.39 325 -0.15 875 

Random Effects 

Level Variance Components Variance 

School Level 2 860.43 

Student Level 1 22,582.88 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 4,163.13 .088 

Student Level 1 42,954.04 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D9 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Letter Name Accuracy Scores 


Level Effect Impact (f) SE DF t p 

School Intercept 98.38 0.30 9 332.37 .000 

Student System 44 0.27 0.51 286 0.52 .604 
Fall SPI Letter Name -0.01 0.02 286 -0.30 764 
Accuracy Score 
Sex 0.93 0.52 286 1.78 .077 
Caucasian -0.80 0.55 286 -1.47 .142 
Special Education -0.37 0.63 286 -0.58 559 
Elementary school level -0.49 0.67 286 -0.74 461 

Random Effects 

Level Variance Components Variance 

School Level 2 0.19 

Student Level 1 19.09 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 0.19 .010 

Student Level 1 19.09 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D10 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Sight Word Accuracy Scores 


Level Effect Impact (f) SE DF t p 

School Intercept 23.94 0.15 9 160.59 .000 

Student System 44 0.03 0.30 286 0.11 .916 
Fall SPI Sight Word 0.53 0.04 286 13.35 .000 
Accuracy Score 
Sex 0.54 0.31 286 1.77 .077 
Caucasian -0.46 0.31 286 -1.45 147 
Special Education -0.28 0.37 286 -0.74 457 
Elementary school level 0.15 0.34 286 0.44 .662 

Random Effects 

Level Variance Components Variance 

School Level 2 0.00 

Student Level 1 6.46 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 .00 .000 

Student Level 1 10.97 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D11 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Sight Word Fluency Scores 


Level Effect Impact (f) SE DF t p 

School Intercept 10.27 0.28 9 37.18 .000 

Student System 44 1.74 0.50 286 3.48 .001 
Fall SPI Sight Word 0.48 0.08 286 6.17 .000 
Fluency Score 
Sex 0.79 0.51 286 1.55 121 
Caucasian -0.92 0.53 286 -1.74 .082 
Special Education -1.28 0.62 286 -2.07 .039 
Elementary school level -2.22 0.62 286 -3.56 001 

Random Effects 

Level Variance Components Variance 

School Level 2 0.12 

Student Level 1 18.16 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 1.05 .046 

Student Level 1 21.63 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D12 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Nonsense Word Accuracy Scores 


Level Effect 
School Intercept 
Student System 44 


Fall SPI Nonsense Word 
Accuracy Score 


Sex 
Caucasian 
Special Education 


Elementary school level 


Fixed Effects 


Impact (f) SE 
24.02 0.21 
2.00 0.37 
0.34 0.05 
0.18 0.38 
0.42 0.40 
-1.11 0.47 
0.35 0.48 


9 
286 
286 


286 
286 
286 
286 


t p 

112.94 .000 
5.36 .000 
6.86 .000 
0.47 .640 
1.07 .287 
-2.37 .018 
0.72 .470 


Random Effects 


Level Variance Components Variance 
School Level 2 0.09 
Student Level 1 10.13 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance Icc 
School Level 2 0.01 .001 
Student Level 1 13.34 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D13 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Nonsense Word Fluency Scores 


Fixed Effects 


Level Effect Impact (f) 

School Intercept 10.27 

Student System 44 1.74 
Fall SPI Nonsense Word 0.48 
Fluency Score 
Sex 0.79 
Caucasian -0.92 
Special Education -1.28 
Elementary school level -2.22 


SE 
0.28 
0.50 
0.08 


0.51 
0.53 
0.62 
0.62 


Random Effects 


Level Variance Components Variance 
School Level 2 1.85 
Student Level 1 20.66 
Random Effects (From Unconditional Model)* 
Level Variance Components Variance Icc 
School Level 2 0.60 021 
Student Level 1 27.45 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D14 


Final Model for Estimating Fixed and Random Effects of System 44 


on Spring SPI Overall Accuracy Scores 


Level Effect 
School Intercept 
Student System 44 


Fall SPI Overall Accuracy 


Score 

Sex 

Caucasian 
Special Education 


Elementary school level 


Fixed Effects 


Impact (f) SE 
47.96 0.28 
1.99 0.56 
0.50 0.05 
0.75 0.57 
-0.13 0.59 
-1.14 0.71 
0.55 0.65 


9 
286 
286 


286 
286 
286 
286 


t p 
170.78 .000 
3.56 .001 
11.00 .000 
1.31 193 
-0.21 .832 
-1.61 107 
0.85 395 


Random Effects 


Level Variance Components Variance 
School Level 2 0.01 
Student Level 1 22.81 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance ICC 
School Level 2 0.00 .000 
Student Level 1 34.84 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D15 


Fixed Effects 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Overall Fluency Scores 


Level Effect Impact (f) SE DF t p 

School Intercept 20.68 0.73 9 28.22 .000 

Student System 44 4.70 0.88 286 5.36 .000 
Fall SPI Overall Fluency 0.69 0.09 286 8.08 .000 
Score 
Sex 0.95 0.90 286 1.06 .288 
Caucasian -1.32 0.96 286 -1.37 172 
Special Education -2.20 1.09 286 -2.02 .045 
Elementary school level -3.30 1.62 286 -2.04 041 

Random Effects 

Level Variance Components Variance 

School Level 2 3.17 

Student Level 1 55.56 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance Icc 

School Level 2 1.68 .022 

Student Level 1 75.18 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 


RMC Research Corporation@Portland, OR 


107 


Appendix E 
Student Proficiency on CST 


Exhibit E1 
Percentage of Students Performing at Each Level on CST 


Program Group Far Below Below Basic Proficient Advanced 
Treatment 
Pretest 8% 18% 64% 10% 1% 
Posttest 4% 14% 41% 32% 9% 
Control 
Pretest 9% 24% 56% 11% 1% 
Posttest 7% 12% 48% 23% 9% 


Note. Treatment total n = 140. Control total n = 147. 


RMC Research Corporation @Portland, OR 


Exhibit E2 
Percentage of Students Achieving Proficiency on CST By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest n Pretest Posttest 
Grade 
4 66 17% 61% 67 21% 51% 
5 22 14% 27% 24 12% 29% 
6 30 3% 13% 30 0% 13% 
7 15 0% 40% 16 0% 6% 
8 7 0% 14% 10 0% 10% 
Decoding Status 
Pre decoder 6 0% 33% 7 29% 43% 
Beginning decoder 26 0% 35% 26 8% 35% 
Developing decoder 105 13% 43% 110 12% 30% 
FRL Status 
None 79 8% 43% 89 16% 39% 
Reduced-Price 46 15% 37% 47 2% 17% 
Free 15 13% 40% 11 18% 36% 
ELL Status 
English only 94 6% 42% 107 14% 36% 
Initial full English speaker 19 21% 58% 12 8% 25% 
English learner 14 0% 14% 17 0% 6% 
Redesignated former English 13 39% 38% 11 9% 46% 
learner 
SPED Status 
None 107 13% 38% 110 14% 33% 
Specific learning disability 8 0% 38% 16 0% 25% 
Speech or language impairment 12 8% 67% 12 17% 42% 
Other classification 13 0% 38% 9 0% 22% 
Ethnicity 
Caucasian 89 11% 43% 93 15% 33% 
African American 11 0% 46% 16 12% 31% 
Hispanic 21 10% 29% 23 0% 30% 
Asian/Pacific Islander 8 12% 25% 7 0% 29% 
Filipino 10 20% 50% 7 14% 29% 

Exhibit continues 
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Exhibit E2 (continued) 


Treatment Control 

Subgroup n Pretest Posttest n Pretest Posttest 
School 

Alta Murrieta 5 0% 40% 5 0% 0% 
Avaxat 15 40% 73% 14 50% 64% 
Buchanan 14 14% 71% 14 7% 57% 
Curran 6 0% 17% 7 0% 29% 
Monte Vista 21 24% 62% 26 35% 58% 
Murrieta 13 0% 23% 12 0% 25% 
Rail Ranch 14 7% 43% 13 0% 31% 
Shivela 17 0% 18% 20 0% 0% 
Thompson 19 0% 26% 18 0% 17% 
McElhinney 4 0% 25% 5 0% 20% 
Warm Springs 12 8% 17% 13 0% 15% 
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Appendix F 
Mean Scores on Reading Tests, SRI, and SPI 
Overall Accuracy and Fluency Outcomes 


Exhibit F1 
TOSREC Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 73 34.48 37.86 3.38 72 34.16 38.59 4.44 
5 25 27.39 32.32 4.93 27 26.25 32.91 6.66 
6 43 30.57 26.83 -3.74 39 24.15 25.81 1.66 
7 20 24.61 28.95 4.34 20 29.21 36.46 7.25 
8 11 13.60 11.08 -2.52 10 12.57 14.45 1.87 
Decoding Status 
Pre decoder 6 35.85 39.05 3.20 7 29.66 26.31 -3.34 
Beginning decoder 44 19.09 21.18 2.09 36 21.59 25.53 3.94 
Developing decoder 119 33.72 34.77 1.05 120 30.85 35.72 4.87 
FRL Status 
None 96 29.65 31.44 1.79 102 30.15 32.81 2.66 
Reduced-Price 58 28.68 29.67 0.98 54 24.45 31.18 6.73 
Free 18 35.14 38.21 3.07 11 37.87 44.43 6.55 
ELL Status 
English only 113 31.34 32.08 0.74 121 29.94 33.52 3.58 
Initial full English speaker 21 28.86 32.94 4.09 14 33.47 38.59 5.12 
English learner 25 17.91 20.72 2.80 20 20.18 28.46 8.28 
Redesignated former 13 41.94 45.50 3.56 12 25.49 29.36 3.87 
English learner student 
SPED Status 
None 120 33.79 35.77 1.97 121 32.44 36.80 4.36 
Specific learning disability 16 17.61 20.26 2.64 20 16.60 20.93 4.33 
Speech or language 12 32.54 34.80 2.26 12 24.43 28.18 3.74 
impairment 
Other classification 24 17.78 16.37 -1.41 15 18.05 22.02 3.97 
Ethnicity 
Caucasian 106 30.69 32.68 1.99 107 30.67 34.32 3.65 
African American 15 30.90 29.95 -0.95 18 28.46 31.63 3.17 
Hispanic 29 29.05 31.18 2.13 25 23.42 31.07 7.64 
Asian/Pacific Islander 10 27.34 28.75 1.41 9 25.07 26.09 1.02 
Filipino 11 27.44 28.58 1.15 7 28.61 37.57 8.96 


Exhibit continues 


RMC Research Corporation @Portland, OR 117 


Exhibit F1 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 22.76 24.42 1.66 5 31.40 20.04 -11.36 
Avaxat 17 37.47 40.43 2.96 14 39.81 40.87 1.06 
Buchanan 15 37.25 39.43 2.19 15 37.63 45.92 8.29 
Curran 8 22.24 25.09 2.85 7 21.39 23.44 2.06 
Monte Vista 25 35.45 39.69 4.24 29 31.03 37.11 6.09 
Murrieta 13 28.33 35.69 7.36 14 32.62 35.98 3.36 
Rail Ranch 15 30.66 34.28 3.62 15 25.54 37.47 11.93 
Shivela 29 26.46 25.27 -1.19 25 25.53 26.54 1.01 
Thompson 23 27.89 25.27 -2.62 18 25.28 28.63 3.35 
McElhinney 6 17.32 16.55 -0.77 9 16.74 19.28 2.53 
Warm Springs 16 26.83 27.59 0.76 17 23.49 30.23 6.74 
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Exhibit F2 
CTOPP Elision Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 73 23.67 25.82 2.15 72 23.29 24.15 0.86 
5 25 22.00 23.20 1.20 27 22.07 24.00 1.93 
6 43 21.70 23.21 1.51 39 22.49 22.56 0.08 
7 20 19.25 20.65 1.40 21 22.00 23.29 1.29 
8 12 23.00 22.50 -0.50 12 20.83 22.25 1.42 
Decoding Status 
Pre decoder 6 22.50 21.83 -0.67 7 20.71 23.57 2.86 
Beginning decoder 45 20.84 22.40 1.56 38 19.24 20.39 1.16 
Developing decoder 119 23.13 24.76 1.63 121 23.65 24.33 0.68 
FRL Status 
None 97 22.13 24.14 2.01 103 22.47 22.97 0.50 
Reduced-Price 58 21.98 23.93 1.95 56 22.23 23.86 1.62 
Free 18 25.00 23.11 -1.89 11 24.82 26.64 1.82 
ELL Status 
English only 114 22.52 24.55 2.04 124 22.24 23.39 1.15 
Initial full English speaker 21 22.86 22.81 -0.05 14 25.71 24.79 -0.93 
English learner 25 19.96 20.04 0.08 20 21.05 22.55 1.50 
Redesignated former 13 25.08 28.23 3.15 12 24.42 24.75 0.33 
English learner student 
SPED Status 
None 120 23.68 25.22 1.54 121 23.80 24.62 0.82 
Specific learning disability 17 18.18 19.41 1.24 21 19.05 20.67 1.62 
Speech or language 12 22.33 23.67 1.33 13 21.23 21.08 -0.15 
impairment 
Other classification 24 18.92 21.08 2.17 16 19.13 21.00 1.88 
Ethnicity 
Caucasian 107 22.43 24.23 1.80 109 22.94 23.26 0.32 
African American 15 22.33 24.20 1.87 18 22.11 23.67 1.56 
Hispanic 29 22.10 23.21 1.10 26 22.54 25.19 2.65 
Asian/Pacific Islander 10 22.30 24.50 2.20 9 18.00 21.44 3.44 
Filipino 11 23.18 23.18 0.00 7 24.00 24.14 0.14 
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Exhibit F2 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 18.20 23.00 4.80 5 23.80 25.00 1.20 
Avaxat 17 23.47 25.18 1.71 14 22.71 22.07 -0.64 
Buchanan 15 24.33 27.40 3.07 15 21.93 24.07 2.13 
Curran 8 23.13 25.75 2.63 7 21.29 25.43 4.14 
Monte Vista 25 22.16 25.76 3.60 29 23.31 24.45 1.14 
Murrieta 13 27.08 23.00 -4.08 14 26.79 24.50 -2.29 
Rail Ranch 15 22.13 24.13 2.00 15 20.47 24.13 3.67 
Shivela 29 22.28 22.66 0.38 25 22.36 21.84 -0.52 
Thompson 24 19.42 21.96 2.54 21 22.38 23.38 1.00 
McElhinney 6 17.50 18.00 0.50 9 20.22 17.22 -3.00 
Warm Springs 16 23.56 24.31 0.75 17 22.24 26.12 3.88 
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Exhibit F3 
Woodcock-Johnson II Word Identification Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 73 50.88 58.47 7.59 72 50.47 57.38 6.90 
5 23 41.52 49.84 8.32 27 43.30 51.63 8.33 
6 42 38.29 46.86 8.57 38 33.18 39.50 6.32 
7 20 34.85 36.90 2.05 20 35.57 42.25 6.68 
8 12 20.17 26.92 6.75 11 22.92 26.55 3.63 
Decoding Status 
Pre decoder 6 39.83 44.17 4.33 7 46.71 52.00 5.29 
Beginning decoder 43 33.28 39.69 6.41 37 31.08 38.30 7.22 
Developing decoder 118 45.83 53.55 7.72 119 44.40 51.26 6.86 
FRL Status 
None 94 43.20 49.88 6.67 102 41.45 47.59 6.14 
Reduced-Price 58 40.60 48.57 7.97 54 39.98 48.52 8.54 
Free 18 44.44 51.94 7.50 11 52.18 59.18 7.00 
ELL Status 
English only 113 42.78 49.83 7.05 121 40.73 47.36 6.64 
Initial full English speaker 19 47.16 53.90 6.75 14 51.00 57.07 6.07 
English learner 25 33.36 40.96 7.60 20 39.20 48.05 8.85 
Redesignated former 13 50.15 57.92 7.77 12 44.50 52.83 8.33 
English learner student 
SPED Status 
None 119 47.47 55.53 8.05 120 46.94 54.17 7.22 
Specific learning disability 16 24.50 32.88 8.38 19 25.76 30.89 5.13 
Speech or language 12 40.25 46.83 6.58 13 33.46 38.23 4.77 
impairment 
Other classification 23 30.09 33.58 3.50 16 28.94 36.19 7.25 
Ethnicity 
Caucasian 104 42.55 49.61 7.06 109 41.82 48.82 7.00 
African American 15 42.53 48.87 6.33 18 41.33 50.06 8.72 
Hispanic 29 40.59 46.45 5.86 24 43.65 49.25 5.60 
Asian/Pacific Islander 10 46.30 53.20 6.90 9 35.44 45.22 9.78 
Filipino 11 44.27 57.36 13.09 6 46.00 50.83 4.83 
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Exhibit F3 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 36.20 41.20 5.00 5 49.60 55.40 5.80 
Avaxat 17 51.88 61.24 9.35 14 50.00 56.21 6.21 
Buchanan 15 48.80 61.33 12.53 15 52.87 62.20 9.33 
Curran 8 50.13 55.75 5.63 7 37.57 48.86 11.29 
Monte Vista 23 48.04 55.36 7.32 29 48.52 55.76 7.24 
Murrieta 13 51.08 53.69 2.62 14 48.71 54.43 5.71 
Rail Ranch 15 46.93 54.60 7.67 15 47.33 53.80 6.47 
Shivela 28 38.00 44.62 6.62 25 32.40 40.64 8.24 
Thompson 24 31.75 35.21 3.46 18 32.76 31.33 -1.43 
McElhinney 6 27.83 41.83 14.00 9 27.78 39.00 11.22 
Warm Springs 16 34.63 42.88 8.25 17 33.41 41.59 8.18 
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Exhibit F4 
Woodcock-Johnson Ill Word Attack Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Posttest Gain 
Grade 
4 73 52.16 56.95 4.78 71 51.36 53.94 2.58 
5 23 47.00 50.76 3.76 27 43.59 47.41 3.81 
6 43 38.49 45.28 6.79 39 35.62 41.87 6.26 
7 20 35.10 37.00 1.90 21 36.19 43.86 7.67 
8 12 25.08 33.83 8.75 12 28.67 32.17 3.50 
Decoding Status 
Pre decoder 6 37.83 47.83 10.00 7 40.86 45.57 4.71 
Beginning decoder 45 35.02 41.44 6.42 37 36.24 40.95 4.71 
Developing decoder 117 47.77 52.19 4.42 121 45.05 49.22 4.17 
FRL Status 
None 96 44.08 49.73 5.65 103 42.04 44.96 2.92 
Reduced-Price 57 42.96 47.83 4.86 55 42.77 49.96 7.20 
Free 18 48.11 51.17 3.06 11 54.18 56.73 2.55 
ELL Status 
English only 113 44.06 49.45 5.39 123 41.93 45.88 3.95 
Initial full English speaker 20 46.85 50.81 3.96 14 46.21 53.07 6.86 
English learner 25 37.88 41.68 3.80 20 41.50 47.60 6.10 
Redesignated former 13 52.62 59.46 6.85 12 53.75 55.42 1.67 
English learner student 
SPED Status 
None 118 48.64 54.18 5.54 120 47.45 51.01 3.56 
Specific learning disability 17 31.29 34.65 3.35 21 27.86 36.43 8.57 
Speech or language 12 42.92 46.58 3.67 13 35.69 40.00 4.31 
impairment 
Other classification 24 31.71 36.25 4.54 16 36.13 40.25 4.13 
Ethnicity 
Caucasian 105 43.25 48.89 5.64 109 43.61 46.90 3.28 
African American 15 44.27 47.13 2.87 17 42.39 47.12 4.73 
Hispanic 29 44.97 46.93 1.97 26 44.46 51.50 7.04 
Asian/Pacific Islander 10 46.70 54.10 7.40 9 35.56 40.78 5.22 
Filipino 11 48.55 58.45 9.91 7 42.71 51.43 8.71 
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Exhibit F4 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 4 37.50 42.40 4.90 5 49.00 57.20 8.20 
Avaxat 17 55.94 58.06 2.12 14 50.79 50.43 -0.36 
Buchanan 15 53.40 56.13 2.73 14 50.07 53.07 3.00 
Curran 8 51.50 58.75 7.25 7 42.00 55.14 13.14 
Monte Vista 24 48.88 54.28 5.41 29 47.79 50.93 3.14 
Murrieta 13 52.31 56.46 4.15 14 50.86 53.07 2.21 
Rail Ranch 15 48.13 54.93 6.80 15 51.73 51.27 -0.47 
Shivela 29 37.34 44.55 7.21 25 37.52 42.44 4.92 
Thompson 24 33.67 34.58 0.92 21 35.29 37.24 1.95 
McElhinney 6 27.17 41.83 14.67 9 28.11 39.56 11.44 
Warm Springs 16 37.75 45.00 7.25 17 33.00 43.59 10.59 
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Exhibit F5 
TOWRE Sigh Word Efficiency Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 73 61.97 67.10 5.14 72 61.99 67.00 5.01 
5 25 63.50 66.48 2.98 27 61.46 65.52 4.06 
6 43 63.58 68.58 5.00 39 62.71 66.31 3.60 
7 20 63.78 67.25 3.47 21 66.07 70.48 4.40 
8 12 58.50 62.42 3.92 12 59.54 64.00 4.46 
Decoding Status 
Pre decoder 6 64.83 68.08 3.25 7 55.71 64.36 8.64 
Beginning decoder 45 54.50 59.80 5.30 38 54.41 57.95 3.54 
Developing decoder 119 65.37 69.68 4.31 121 65.05 69.58 4.53 
FRL Status 
None 97 61.46 66.32 4.86 103 62.75 66.40 3.65 
Reduced-Price 58 63.71 67.76 4.05 56 61.71 67.60 5.88 
Free 18 64.75 68.92 4.17 11 62.91 67.73 4.82 
ELL Status 
English only 114 62.68 67.44 4.75 124 62.35 66.45 4.10 
Initial full English speaker 21 62.69 65.88 3.19 14 65.96 71.07 5.11 
English learner 25 58.82 63.24 4.42 20 60.58 66.25 5.68 
Redesignated former 13 68.42 73.15 4.73 12 62.04 67.50 5.46 
English learner student 
SPED Status 
None 120 64.62 69.13 4.51 121 64.86 69.21 4.35 
Specific learning disability 17 55.94 60.94 5.00 21 57.45 63.29 5.83 
Speech or language 12 64.83 68.67 3.83 13 56.31 60.85 4.54 
impairment 
Other classification 24 55.81 60.35 4.54 16 55.25 58.28 3.03 
Ethnicity 
Caucasian 107 62.68 67.11 4.43 109 62.37 66.24 3.88 
African American 15 62.20 68.57 6.37 18 65.03 68.47 3.44 
Hispanic 29 63.16 66.57 3.41 26 60.69 67.25 6.56 
Asian/Pacific Islander 10 62.45 67.80 5.35 9 56.56 64.17 7.61 
Filipino 11 61.95 66.14 4.18 7 70.36 75.00 4.64 
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Exhibit F5 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 60.40 66.00 5.60 5 56.30 64.40 8.10 
Avaxat 17 61.18 68.35 7.18 14 61.29 66.36 5.07 
Buchanan 15 65.63 69.23 3.60 15 66.17 70.67 4.50 
Curran 8 60.81 64.00 3.19 7 57.14 62.21 5.07 
Monte Vista 25 62.56 67.14 4.58 29 63.69 67.60 3.91 
Murrieta 13 62.77 63.69 0.92 14 60.57 63.54 2.96 
Rail Ranch 15 61.20 67.43 6.23 15 59.73 66.43 6.70 
Shivela 29 64.78 69.71 4.93 25 65.86 69.66 3.80 
Thompson 24 60.33 63.35 3.02 21 62.19 65.02 2.83 
McElhinney 6 52.00 57.92 5.92 9 60.94 65.11 4.17 
Warm Springs 16 67.06 72.09 5.03 17 61.56 67.12 5.56 
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Exhibit F6 
TOWRE Phonetic Decoding Efficiency Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 73 30.33 34.10 3.77 72 29.95 34.06 4.11 
5 25 31.70 34.26 2.56 27 30.24 31.78 1.54 
6 43 30.26 34.90 4.64 39 27.91 32.51 4.60 
7 20 29.65 32.75 3.10 21 34.83 39.76 4.93 
8 12 26.83 29.33 2.50 12 29.96 32.75 2.79 
Decoding Status 
Pre decoder 6 29.17 33.00 3.83 7 23.00 28.00 5.00 
Beginning decoder 45 23.57 27.20 3.63 38 22.45 27.28 4.83 
Developing decoder 119 32.60 36.27 3.67 121 32.43 36.10 3.68 
FRL Status 
None 97 29.14 33.28 4.14 103 29.23 33.27 4.04 
Reduced-Price 58 31.34 34.24 2.91 56 30.63 34.85 4.21 
Free 18 32.14 35.47 3.33 11 35.05 35.64 0.59 
ELL Status 
English only 114 29.77 33.60 3.83 124 29.42 33.31 3.89 
Initial full English speaker 21 31.86 35.81 3.95 14 36.00 39.32 3.32 
English learner 25 27.14 30.08 2.94 20 29.18 34.25 5.08 
Redesignated former 13 37.04 39.88 2.85 12 31.33 33.75 2.42 
English learner student 
SPED Status 
None 120 31.90 35.58 3.67 121 32.26 35.69 3.43 
Specific learning disability 17 24.68 29.62 4.94 21 22.76 29.14 6.38 
Speech or language 12 32.29 33.96 1.67 13 23.73 26.69 2.96 
impairment 
Other classification 24 24.46 28.02 3.56 16 28.88 33.06 4.19 
Ethnicity 
Caucasian 107 29.69 33.53 3.85 109 29.56 33.27 3.71 
African American 15 29.87 34.07 4.20 18 31.64 34.69 3.06 
Hispanic 29 31.93 33.66 1.72 26 31.04 35.12 4.08 
Asian/Pacific Islander 10 29.45 34.70 5.25 9 22.00 29.11 7.14 
Filipino 11 33.05 37.23 4.18 7 40.64 43.79 3.14 
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Exhibit F6 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 26.80 24.90 -1.90 5 31.70 31.90 0.20 
Avaxat 17 29.82 34.38 4.56 14 29.75 31.68 1.93 
Buchanan 15 32.13 36.80 4.67 15 33.07 36.17 3.10 
Curran 8 27.44 29.69 2.25 7 28.21 27.79 -0.43 
Monte Vista 25 32.50 37.18 4.68 29 30.43 35.43 5.00 
Murrieta 13 30.04 32.46 2.42 14 26.86 30.93 4.07 
Rail Ranch 15 30.73 33.03 2.30 15 29.73 34.00 4.27 
Shivela 29 31.10 35.59 4.48 25 31.84 36.48 4.64 
Thompson 24 27.92 30.21 2.29 21 31.69 35.21 3.52 
McElhinney 6 24.33 29.58 5.25 9 27.06 32.61 5.56 
Warm Springs 16 31.13 35.81 4.69 17 27.91 32.41 4.50 
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Exhibit F7 
SRI Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 73 411.21 586.96 175.75 71 392.50 541.96 149.46 
5 25 375.96 584.44 208.48 27 369.67 557.44 187.78 
6 43 413.47 579.53 166.07 33 385.81 478.39 92.58 
7 20 433.70 646.60 212.90 20 442.33 647.65 205.32 
8 12 340.92 497.83 156.92 8 457.83 477.63 19.79 
Decoding Status 
Pre decoder 6 405.00 545.33 140.33 7 489.57 543.86 54.29 
Beginning decoder 45 305.18 485.58 180.40 34 310.89 442.24 131.34 
Developing decoder 119 440.87 625.94 185.08 114 419.08 570.44 151.36 
FRL Status 
None 97 391.45 585.71 194.26 96 417.14 562.33 145.20 
Reduced-Price 58 434.84 594.57 159.72 52 357.82 490.19 132.37 
Free 18 376.06 554.78 178.72 11 428.45 601.55 173.09 
ELL Status 
English only 114 409.70 597.31 187.61 114 417.93 566.18 148.25 
Initial full English speaker 21 445.71 631.29 185.57 14 469.29 600.71 131.43 
English learner 25 285.12 444.60 159.48 19 202.80 339.26 136.46 
Redesignated former 13 520.54 678.46 157.92 12 438.92 557.50 118.58 
English learner student 
SPED Status 
None 120 425.58 626.53 200.95 120 407.88 563.46 155.57 
Specific learning disability 17 361.88 477.35 115.47 16 377.24 478.88 101.64 
Speech or language 12 397.92 602.08 204.17 11 377.75 478.91 101.16 
impairment 
Other classification 24 331.88 448.42 116.54 12 366.00 462.17 96.17 
Ethnicity 
Caucasian 107 418.11 598.56 180.45 102 421.75 562.60 140.85 
African American 15 403.20 577.40 174.20 18 429.56 510.17 80.61 
Hispanic 29 393.03 579.90 186.86 24 299.65 471.17 171.51 
Asian/Pacific Islander 10 323.10 523.10 200.00 8 366.56 522.50 155.94 
Filipino 11 392.09 568.27 176.18 7 379.57 576.43 196.86 
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Exhibit F7 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 232.40 398.60 166.20 5 246.20 300.60 54.40 
Avaxat 17 478.82 682.35 203.53 14 494.50 623.93 129.43 
Buchanan 15 468.60 642.87 174.27 15 523.27 636.60 113.33 
Curran 8 300.25 561.38 261.13 7 164.43 395.43 231.00 
Monte Vista 25 407.12 578.92 171.80 28 413.66 568.29 154.63 
Murrieta 13 351.38 492.31 140.92 14 262.79 435.64 172.86 
Rail Ranch 15 395.87 590.60 194.73 15 360.80 597.60 236.80 
Shivela 29 411.93 571.03 159.10 23 449.46 500.22 50.76 
Thompson 24 373.58 586.58 213.00 12 380.00 607.67 227.67 
McElhinney 6 381.33 471.50 90.17 9 410.78 535.44 124.67 
Warm Springs 16 459.00 647.44 188.44 17 410.24 526.18 115.94 
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Exhibit F8 
SPI Overall Accuracy Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 72 41.72 49.26 7.54 70 41.21 46.46 5.24 
5 25 41.96 49.56 7.60 26 41.12 47.26 6.14 
6 42 42.86 48.16 5.31 30 42.23 46.05 3.82 
7 19 42.58 47.30 4.72 16 44.69 48.10 3.41 
8 11 44.18 46.67 2.48 6 39.50 43.58 4.08 
Decoding Status 
Pre decoder 6 37.67 47.67 10.00 7 33.71 48.00 14.29 
Beginning decoder 44 39.43 44.73 5.30 35 38.46 43.32 4.86 
Developing decoder 118 43.64 50.20 6.57 106 43.31 47.40 4.09 
FRL Status 
None 97 41.58 48.04 6.46 87 41.48 46.24 4.76 
Reduced-Price 55 43.04 49.22 6.19 52 41.85 46.82 4.98 
Free 17 44.00 49.83 5.83 9 43.11 47.18 4.07 
ELL Status 
English only 110 42.34 48.48 6.15 103 41.55 46.17 4.62 
Initial full English speaker 21 42.86 50.38 7.52 14 43.93 49.36 5.43 
English learner 25 39.16 45.96 6.80 20 40.15 46.00 5.85 
Redesignated former 13 47.08 52.15 5.08 11 43.18 47.33 4.15 
English learner student 
SPED Status 
None 118 42.96 49.76 6.80 111 42.58 47.44 4.86 
Specific learning disability 16 40.00 46.12 6.12 15 41.40 45.43 4.03 
Speech or language 12 41.08 49.42 8.33 10 37.10 43.46 6.36 
impairment 
Other classification 23 41.13 44.33 3.20 12 37.92 43.00 5.08 
Ethnicity 
Caucasian 104 42.10 48.50 6.40 94 41.64 46.30 4.66 
African American 15 42.73 48.80 6.07 15 43.33 47.56 4.22 
Hispanic 28 42.96 48.21 5.24 23 40.00 45.31 5.31 
Asian/Pacific Islander 10 40.60 49.00 8.40 9 39.44 46.56 7.11 
Filipino 11 43.64 50.82 7.18 7 47.71 51.86 4.14 
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Exhibit F8 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 41.20 48.00 6.80 5 40.40 47.00 6.60 
Avaxat 17 42.00 50.35 8.35 14 41.57 47.43 5.86 
Buchanan 15 43.73 49.07 5.33 13 44.54 47.87 3.33 
Curran 8 43.75 51.00 7.25 7 37.00 43.57 6.57 
Monte Vista 24 39.67 48.36 8.69 28 41.43 46.69 5.26 
Murrieta 13 42.00 48.23 6.23 14 39.07 45.86 6.79 
Rail Ranch 15 41.93 50.60 8.67 15 41.67 46.87 5.20 
Shivela 28 43.25 48.28 5.03 24 44.00 47.29 3.29 
Thompson 22 43.32 46.67 3.35 2 36.00 43.81 7.81 
McElhinney 6 39.83 43.67 3.83 9 43.11 46.44 3.33 
Warm Springs 16 43.25 49.69 6.44 17 41.35 47.65 6.29 
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Exhibit F9 
SPI Overall Fluency Means By Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 72 12.97 22.12 9.15 70 13.99 18.49 4.50 
5 25 15.00 22.40 7.40 26 13.73 18.11 4.38 
6 42 14.36 23.81 9.46 30 13.93 18.50 4.57 
7 19 14.16 22.10 7.94 16 16.13 21.71 5.59 
8 11 11.64 20.00 8.36 6 13.00 19.75 6.75 
Decoding Status 
Pre decoder 6 8.17 24.50 16.33 7 6.86 11.86 5.00 
Beginning decoder 44 6.77 15.49 8.72 35 7.69 13.24 5.55 
Developing decoder 118 16.43 25.13 8.70 106 16.73 20.90 4.17 
FRL Status 
None 97 13.26 22.02 8.76 87 13.60 19.24 5.65 
Reduced-Price 55 13.95 21.66 7.71 52 14.46 18.54 4.07 
Free 17 15.06 27.17 12.11 9 17.22 17.82 0.60 
ELL Status 
English only 110 13.59 21.72 8.13 103 13.80 18.40 4.60 
Initial full English speaker 21 15.33 26.76 11.43 14 15.50 23.43 7.93 
English learner 25 11.16 20.64 9.48 20 13.30 18.75 5.45 
Redesignated former 13 16.38 25.15 8.77 11 16.91 19.33 2.42 
English learner student 
SPED Status 
None 118 14.26 23.47 9.20 111 14.52 20.17 5.65 
Specific learning disability 16 11.56 17.41 5.85 15 13.07 17.10 4.03 
Speech or language 12 13.92 26.67 12.75 10 14.60 15.23 0.63 
impairment 
Other classification 23 11.91 18.71 6.80 12 11.33 14.53 3.20 
Ethnicity 
Caucasian 104 13.75 22.13 8.38 94 13.95 18.40 4.46 
African American 15 12.80 22.93 10.13 15 15.00 21.28 6.28 
Hispanic 28 13.54 21.62 8.08 23 14.48 18.31 3.83 
Asian/Pacific Islander 10 12.70 24.60 11.90 9 13.22 15.11 1.89 
Filipino 11 15.00 25.55 10.55 7 14.57 27.57 13.00 
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Exhibit F9 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
School 

Alta Murrieta 5 11.20 14.20 3.00 5 15.00 15.80 0.80 
Avaxat 17 13.59 21.76 8.18 14 16.00 18.36 2.36 
Buchanan 15 13.40 21.33 7.93 13 15.92 18.07 2.14 
Curran 8 15.38 19.25 3.88 7 12.43 12.86 0.43 
Monte Vista 24 12.71 24.60 11.89 28 13.36 20.66 7.30 
Murrieta 13 14.46 23.00 8.54 14 12.07 15.64 3.57 
Rail Ranch 15 13.67 23.07 9.40 15 13.33 20.33 7.00 
Shivela 28 13.14 22.45 9.31 24 15.04 20.83 5.79 
Thompson 22 14.68 20.00 5.32 2 12.00 18.33 6.33 
McElhinney 6 11.67 23.00 11.33 9 14.67 22.00 7.33 
Warm Springs 16 14.94 27.31 12.38 17 13.94 18.41 4.47 
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Appendix G 
Student Characteristics 


Exhibit G1 
Characteristics of Students By Exit Date 


Exit Date 
Winter Spring End-of-Year 

Characteristic (n= 45) (n= 56) (n=71) 
Grade Level 

4 12 (27%) 32 (57%) 28 (39%) 

5 5 (11%) 11 (20%) 9 (13%) 

6 14 (381%) 11 (20%) 18 (25%) 

7 10 (22%) 1 (2%) 9 (13%) 

8 4 (9%) 1 (2%) 7 (10%) 
Sex 

Male 30 (67%) 28 (50%) 48 (68%) 

Female 15 (33%) 28 (50%) 23 (32%) 
Free or Reduced-Price Meals 

None 26 (58%) 29 (52%) 41 (58%) 

Reduced-price 14 (81%) 19 (84%) 25 (35%) 

Free 5 (11%) 8 (14%) 5 (7%) 
English Learner Status 

English only 32 (71%) 33 (59%) 49 (69%) 

Initial full English speaker 6 (13%) 8 (14%) 6 (8%) 

English learner 2 (4%) 8 (14%) 15 (21%) 

Redesignated former English 5 (11%) 7 (13%) 1 (1%) 

learner student 
Special Education Status 

None 33 (73%) 47 (84%) 39 (55%) 

Specific learning disability 2 (4%) 3 (5%) 12 (17%) 

Speech or language impairment 4 (9%) 5 (9%) 3 (4%) 

Other classification 6 (13%) 1 (2%) 17 (24%) 
Primary Ethnicity 

Caucasian 31 (69%) 33 (59%) 43 (61%) 

African American 2 (4%) 6 (11%) 7 (10%) 

Hispanic 7 (16%) 11 (20%) 11 (16%) 

Asian/Pacific Islander 1 (2%) 3 (5%) 5 (7%) 

Filipino 4 (9%) 3 (5%) 4 (6%) 

Exhibit continues 
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Exhibit G1 (continued) 


Exit Date 
Winter Spring End-of-Year 
Characteristic (n= 45) (n= 56) (n= 71) 
School 
Alta Murrieta 0 (0%) 5 (9%) 0 (0%) 
Avaxat 2 (4%) 7 (13%) 8 (11%) 
Buchanan 0 (0%) 14 (25%) 1 (1%) 
Curran 0 (0%) 2 (4%) 6 (8%) 
Monte Vista 5 (11%) 7 (13%) 12 (17%) 
Murrieta 4 (9%) 5 (9%) 4 (6%) 
Rail Ranch 6 (13%) 3 (5%) 6 (8%) 
Shivela 11 (24%) 8 (14%) 10 (14%) 
Thompson 9 (20%) 3 (5%) 12 (17%) 
McElhinney 2 (4%) 1 (2%) 3 (4%) 
Warm Springs 6 (13%) 1 (2%) 9 (13%) 
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Exhibit G2 
Characteristics of Students By Total Number of Topics Completed 


Number of Completed Topics 


Less Than 100 100 to 159 160 
Characteristic (n= 43) (n= 37) (n= 92) 
Grade Level 
4 21 (49%) 22 (59%) 29 (32%) 
5 6 (14%) 10 (27%) 9 (10%) 
6 7 (16%) 3 (8%) 33 (36%) 
7 6 (14%) 1 (3%) 13 (14%) 
8 3 (7%) 1 (3%) 8 (9%) 
Sex 
Male 27 (63%) 25 (68%) 54 (59%) 
Female 16 (37%) 12 (382%) 38 (41%) 
Free or Reduced-Price Meals 
None 24 (56%) 18 (49%) 54 (59%) 
Reduced-price 15 (85%) 14 (88%) 29 (382%) 
Free 4 (9%) 5 (14%) 9 (10%) 
English Learner Status 
English only 30 (70%) 22 (59%) 62 (67%) 
Initial full English speaker 2 (5%) 6 (16%) 12 (13%) 
English learner 10 (23%) 4 (11%) 11 (12%) 
Redesignated former English 1 (2%) 5 (14%) 7 (8%) 
learner student 
Special Education Status 
None 26 (60%) 29 (78%) 64 (70%) 
Specific learning disability 4 (9%) 5 (14%) 8 (9%) 
Speech or language impairment 3 (7%) 1 (8%) 8 (9%) 
Other classification 10 (23%) 2 (5%) 12 (138%) 
Primary Ethnicity 
Caucasian 26 (62%) 20 (54%) 61 (66%) 
African American 2 (5%) 3 (8%) 10 (11%) 
Hispanic 9 (21%) 7 (19%) 13 (14%) 
Asian/Pacific Islander 4 (10%) 2 (5%) 3 (8%) 
Filipino 1 (2%) 5 (14%) 5 (5%) 
Exhibit continues 
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Exhibit G2 (continued) 


Number of Completed Topics 


Less Than 100 100 to 159 160 
Characteristic (n= 43) (n= 37) (n= 92) 
School 
Alta Murrieta 5 (12%) 0 (0%) 0 (0%) 
Avaxat 4 (9%) 4 (11%) 9 (10%) 
Buchanan 6 (14%) 8 (22%) 1 (1%) 
Curran 4 (9%) 2 (5%) 2 (2%) 
Monte Vista 5 (12%) 10 (27%) 9 (10%) 
Murrieta 2 (5%) 3 (8%) 8 (9%) 
Rail Ranch 1 (2%) 5 (14%) 9 (10%) 
Shivela 3 (7%) 4 (11%) 22 (24%) 
Thompson 10 (23%) 1 (3%) 13 (14%) 
McElhinney 3 (7%) 0 (0%) 3 (3%) 
Warm Springs 0 (0%) 0 (0%) 16 (17%) 
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Exhibit G3 
Characteristics of Students By Initial Decoding Status 


Decoding Status 


Pre Beginning Developing 
Characteristic (n= 6) (n= 44) (n= 118) 
Grade Level 
4 4 (67%) 19 (43%) 48 (41%) 
5 1 (17%) 5 (11%) 19 (16%) 
6 0 (0%) 10 (23%) 2 (27%) 
7 0 (0%) 5 (11%) 4 (12%) 
8 1 (17%) 5 (11%) 5 (4%) 
Sex 
Male 5 (83%) 27 (61%) 72 (61%) 
Female 1 (17%) 17 (39%) 46 (39%) 
Free or Reduced-Price Meals 
None 3 (50%) 31 (70%) 62 (53%) 
Reduced-price 1 (17%) 13 (80%) 41 (85%) 
Free 2 (33%) 0 (0%) 15 (13%) 
English Learner Status 
English only 5 (83%) 30 (68%) 75 (64%) 
Initial full English speaker 0 (0%) 4 (9%) 16 (14%) 
English learner 1 (17%) 9 (20%) 5 (13%) 
Redesignated former English learner 0 (0%) 1 (2%) 12 (10%) 
student 
Special Education Status 
None 6 (100%) 23 (52%) 88 (75%) 
Specific learning disability 0 (0%) 7 (16%) 9 (8%) 
Speech or language impairment 0 (0%) 4 (9%) 8 (7%) 
Other classification 0 (0%) 10 (23%) 13 (11%) 
Primary Ethnicity 
Caucasian 3 (50%) 27 (61%) 74 (63%) 
African American 1 (17%) 4 (9%) 0 (9%) 
Hispanic 1 (17%) 8 (18%) 9 (16%) 
Asian/Pacific Islander 1 (17%) 4 (9%) 4 (8%) 
Filipino 0 (0%) 1 (2%) 10 (9%) 
Exhibit continues 
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Exhibit G3 (continued) 


Decoder Status 


Pre Beginning Developing 
Characteristic (n= 6) (n= 44) (n= 118) 
School 
Alta Murrieta 0 (0%) 2 (5%) 3 (3%) 
Avaxat 0 (0%) 4 (9%) 13 (11%) 
Buchanan 1 (17%) 4 (9%) 10 (8%) 
Curran 0 (0%) 1 (2%) Res %) 
Monte Vista 4 (67%) 6 (14%) 3 (11%) 
Murrieta 0 (0%) 3 (7%) 0 (8%) 
Rail Ranch 0 (0%) 4 (9%) 1 (9%) 
Shivela 0 (0%) 9 (20%) 9 (16%) 
Thompson 1 (17%) 6 (14%) 5 (13%) 
McElhinney 0 (0%) 2 (5%) 4 (3%) 
Warm Springs 0 (0%) 3 (7%) 13 (11%) 
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Appendix H 
Baseline Equivalence Tests 


Exhibit H1 
Baseline Equivalence of Students Exiting in Winter, Spring, and End of Year 
on Outcome Measures and Demographic Characteristics 


A Winter Exits Spring Exits End-of-Year Exits 
Baseline 
Characteristic n M SD n M SD n M SD p 
TOSREC? 45 38.38 15.48 56 33.30 13.84 71 21.93 14.06 .000 
CTOPP Elision® 45 25.51 6.00 56 22.75 6.44 71 20.24 6.17 .000 
Woodcock-Johnson 44 47.86 12.13 56 47.66 13.51 69 34.74 15.74  .000 
Ill Word 
Identification® 
Woodcock-Johnson 44 49.95 13.48 55 49.16 13.86 71 36.69 16.53  .000 
Ill Word Attack 
TOWRE Sight Word 45 69.81 6.93 56 64.29 6.43 71 56.59 10.75  .000 
Efficiency” 
TOWRE Phonetic 45 37.76 8.61 56 32.38 8.81 71 23.69 8.67  .000 
Decoding 
Efficiency” 
CST° 43 322.51 30.79 50 324.42 33.65 48 300.90 39.57 .002 
SRI° 45 498.91 132.58 56 408.14 160.88 71 343.66 184.43 .000 
SPI Letter Name 44 99.59 1.90 55 94.25 13.41 69 95.68 12.87  .060 
Accuracy® 
SPI Sight Word 44 23.77 2.58 55 22.35 3.16 69 19.28 3.80  .000 
Accuracy*® 
SPI Sight Word 44 8.16 3.17 55 6.60 3.42 69 5.67 3.14 .001 
Fluency® 
SPI Nonsense Word 44 23.00 2.81 55 21.47 3.87 69 19.01 3.79  .000 
Accuracy® 
SPI Nonsense Word 44 8.98 3.10 55 7.25 3.09 69 5.49 3.45  .000 
Fluency*® 
SPI Overall 44 46.77 4.44 55 43.82 5.82 69 38.29 6.04 .000 
Accuracy® 
SPI Overall 44 17.14 3.69 55 13.85 4.52 69 11.16 5.67  .000 
Fluency® 
Female 45 0.33 0.48 56 0.50 0.50 71 0.32 0.47 .093 
Black 45 0.04 0.21 56 0.11 0.31 70 0.10 0.30 .489 
Hispanic 45 0.16 0.37 56 0.20 0.40 70 0.16 0.37 .811 
SPED Status 45 0.27 0.45 56 0.16 0.37 71 0.45 0.50 .001 
ELL Status 45 0.29 0.46 56 0.41 0.50 71 0.31 0.47 361 


*Pretest assessment was analyzed using normal curve equivalent scores that ran 


ged from 1 to 99. "Assessment 


was analyzed using raw scores. “Assessment was analyzed using scaled scores. “Assessment was analyzed using 


Lexile scores. 
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Exhibit H2 
Baseline Equivalence of Topic Completion Groups 
on Outcome Measures and Demographic Characteristics 


Completed All Completed Between Completed Fewer 

: 160 Topics 100 and 159 Topics Than 100 Topics 
Baseline 
Characteristic n M SD n M SD n M SD p 
TOSREC? 92 33.51 15.82 37 30.61 14.45 43 21.70 14.56 .000 
CTOPP Elision® 92 23.39 6.58 37 23.76 5.40 43 19.26 6.42 .001 
Woodcock-Johnson 92 44.83 14.84 35 46.43 12.32 42 33.88 16.20 .000 
Ill Word 
Identification* 
Woodcock-Johnson 92 46.32 15.99 36 49.86 13.08 42 34.55 15.13 .000 
Ill Word Attack* 
TOWRE Sight Word 92 66.61 7.67 37 62.85 7.23 43 53.63 11.31 .000 
Efficiency 
TOWRE Phonetic 92 33.76 9.56 37 31.03 8.91 43 21.88 8.86 .000 
Decoding 
Efficiency” 
CST° 79 319.43 29.85 27 =330.78 = 32.38 35 296.17 44.67 .000 
SRI° 92 444.14 158.82 37 456.16 127.88 43 278.33 184.55 .000 
SPI Letter Name 91 97.22 9.71 37 96.35 10.05 40 93.90 15.56 .312 
Accuracy® 
SPI Sight Word 91 22.65 3.34 37 21.30 3.87 40 18.90 3.48 .000 
Accuracy® 
SPI Sight Word 91 7.32 3.60 37 6.65 2.85 40 5.03 2.74 .001 
Fluency*® 
SPI Nonsense Word = 91 21.90 3.52 37 21.27 3.40 40 18.13 4.08 .000 
Accuracy® 
SPI Nonsense Word = 91 7.75 3.31 37 7.49 3.76 40 4.78 2.85 .000 
Fluency® 
SPI Overall 91 44.55 5.58 37 42.57 6.31 40 37.03 6.12 .000 
Accuracy*® 
SPI Overall 91 15.07 4.91 37 14.14 5.20 40 9.80 4.84 .000 
Fluency® 
Female 92 0.41 0.50 37 0.32 0.47 43 0.37 0.49 .638 
Black 92 0.11 0.31 37 0.08 0.28 42 0.05 0.22 .509 
Hispanic 92 0.14 0.35 37 0.19 0.40 42 0.21 0.42 .548 
SPED Status 92 0.30 0.46 37 0.22 0.42 43 0.40 0.49 .225 
ELL Status 92 0.33 0.47 37 0.41 0.50 43 0.30 0.46 .595 


*Pretest assessment was analyzed using normal curve equivalent scores that ranged from 1 to 99. "Assessment 
was analyzed using raw scores. “Assessment was analyzed using scaled scores. “Assessment was analyzed using 
Lexile scores. 
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Exhibit H3 


Baseline Equivalence of Initial Decoding Status Groups 
on Outcome Measures and Demographic Characteristics 


Pre Decoder 


Beginning Decoder 


Developing Decoder 


Baseline 

Characteristic n M SD n M SD n M SD p 
TOSREC? 6 35.85 14.72 44 19.50 13.27 118 33.74 15.14  .000 
CTOPP Elision” 6 22.50 5.24 44 20.73 6.99 118 23.13 6.41 119 
Woodcock-Johnson 6 39.83 12.17 42 34.05 18.90 117 45.90 12.37  .000 
Ill Word 

Identification* 

Woodcock-Johnson 6 37.83 14.08 44 35.73 17.49 116 47.87 14.03 .000 
Ill Word Attack* 

TOWRE Sight Word 6 64.83 7.02 44 54.84 11.74 118 65.34 7.80  .000 
Efficiency” 

TOWRE Phonetic 6 29.17 13.21 44 23.83 10.36 118 32.67 9.28 .000 
Decoding 

Efficiency” 

CST 6 298.33 41.88 28 294.14 41.19 104 323.41 31.21 .000 
SRI° 6 405.00 211.87 44 308.75 188.34 118 440.46 155.43 .000 
SPI Letter Name 6 48.67 16.18 44 98.16 6.01 118 97.94 6.49 .000 
Accuracy® 

SPI Sight Word 6 18.83 3.82 44 19.75 3.62 118 22.23 3.62 .000 
Accuracy*® 

SPI Sight Word 6 3.67 2.16 44 3.36 2.06 118 7.99 2.85  .000 
Fluency® 

SPI Nonsense Word 6 18.83 4.83 44 19.68 4.37 118 21.41 3.61 .019 
Accuracy® 

SPI Nonsense Word 6 4.50 2.26 44 3.41 2.06 118 8.44 2.94  .000 
Fluency® 

SPI Overall 6 37.67 8.04 44 39.43 6.68 118 43.64 6.08  .000 
Accuracy® 

SPI Overall 6 8.17 2.56 44 6.77 2.44 118 16.43 3.43  .000 
Fluency*® 

Female 6 0.17 0.41 44 0.39 0.49 118 0.39 0.49 .550 
Black 6 0.17 0.41 44 0.09 0.29 117 0.09 0.28 .797 
Hispanic 6 0.17 0.41 44 0.18 0.39 117 0.16 0.37 .958 
SPED Status 6 0.00 0.00 44 0.48 0.51 118 0.25 0.44 .006 
ELL Status 6 0.17 0.41 44 0.32 0.47 118 0.36 0.48 .559 


*Pretest assessment was analyzed using normal curve equivalent scores that ran 


ged from 1 to 99. "Assessment 


was analyzed using raw scores. “Assessment was analyzed using scaled scores. “Assessment was analyzed using 


Lexile scores. 
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Appendix | 
System 44 Software Usage Descriptives 


Exhibit 11 
System 44 Software Usage Descriptives 


Number of Number of 
Total Topics Total Hours on | Topics Fast- Median Weeks in Sessions Per 
Total Sessions Completed Software Tracked Session Time System 44 Week 

Subgroup n M SD M SD M SD M SD M SD M SD M SD 
Grade 
4 72 61.14 19.82 | 129.13 43.61 22.72 8.29 | 67.43 39.92 22.76 3.64) 34.61 8.79 1.77 0.33 
5 25 63.12 18.45 | 135.00 37.91 21.92 6.38 | 68.04 37.23 20.32 3.86 34.83 7.96 1.81 0.29 
6 43 70.26 31.32 | 143.21 37.04 | 25.93 11.52 | 68.35 37.13 22.12 6.27) 35.01 12.63 2.03 0.48 
7 20 61.35 31.84 | 126.15 52.33 21.35 9.74 | 68.10 41.92 20.70 6.32) 30.45 14.21 2.11 0.61 
8 12 71.33 34.30 | 127.42 54.59 28.16 16.70 | 52.25 50.91 22.08 6.35) 36.52 14.96 1.97 0.44 
Decoding Status” 
Pre decoder 44 67.50 16.88 | 126.83 41.35 | 26.32 6.49 | 52.00 40.64 > 20.83 549) 37.93 6.83 1.77 0.25 
Beginning decoder 118 77.98 26.94 | 103.36 53.12 29.65 12.86 | 29.18 37.17 20.77 5.18 | 40.71 7.51 1.88 0.41 
Developing decoder 6 58.94 23.62 | 146.19 30.53 21.29 7.91 | 82.93 28.66 22.52 4.85) 31.73 11.06 1.91 0.45 
FRL Status 
None 96 63.76 23.10 133.00 44.02 | 24.11 9.78 | 66.18 40.74 | 22.63 4.83 | 33.82 10.97 1.93 0.43 
Reduced-Price 58 66.69 30.73 | 131.83 43.44 | 23.11 10.95 | 66.34 39.33 20.59 519) 35.74 10.71 1.84 0.45 
Free 18 60.83 20.85 137.11 39.79 | 22.73 7.81 | 71.28 36.54 22.83 4.73 | 33.08 11.52 1.91 0.39 
ELL Status 
English only 114 61.96 23.84 | 131.86 42.97 | 23.30 9.97 | 66.03 39.60 22.39 4.85) 33.72 11.01 1.87 0.42 
Initial full English 20 62.55 21.69 145.65 38.69 | 23.49 8.75 | 78.20 33.79 | 22.95 4.02 | 32.77 10.72 1.97 0.44 
speaker 
English learner 25 82.52 30.24 118.92 50.35 | 28.10 10.95 | 46.76 40.85 | 19.76 5.43 | 40.74 7.62 2.01 0.51 
Redesignated 13 54.31 24.28 151.08 26.58 | 18.16 6.77 | 94.15 25.17 20.92 6.09 | 30.58 12.33 1.81 0.36 


former English 
learner 
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Exhibit 11 (continued) 


Number of Number of 
Total Topics Total Hours on | Topics Fast- Median Weeks in Sessions Per 
Total Sessions Completed Software Tracked Session Time System 44 Week 
Subgroup n M SD M SD M SD M SD M SD M SD M SD 
SPED Status 
None 119 59.90 22.54 | 137.62 38.24 22.16 8.01 73.87 37.72) 22.51 4.43 32.85 10.09 1.85 0.41 
Specific learning 12 86.24 24.66 | 131.24 41.08 32.61 13.06 | 43.88 36.37 19.94 524) 4213 888 2.04 0.36 
disability 


Speech or language 17 51.08 25.05 | 127.33 57.86} 19.65 11.11 | 66.75 4466) 23.17 5.47) 29.15 12.03 1.88 0.73 
impairment 


Other classification 24 78.21 28.49 | 114.42 55.67 | 26.55 12.05) 47.75 38.66) 20.04 659 39.16 12.08 2.01 0.38 


Ethnicity 

Caucasian 107 62.84 23.16 | 133.01 43.37 | 23.75 9.70 | 66.79 39.70 | 2286 4.86 33.65 10.89 1.89 0.38 
African American 15 68.20 29.48 147.53 27.62 | 27.57 13.41 | 69.47 31.99 21.87 4.17) 34.96 10.85 1.94 0.46 
Hispanic 29 67.52 34.44 | 124.79 51.46 | 20.63 8.61 66.10 42.21 18.79 5.54) 36.53 10.82 1.82 0.56 
Asian/ 9 69.56 21.04 | 121.00 46.43 | 26.44 8.43 50.11 49.68 21.00 4.24 > 36.49 11.63 1.96 0.29 
Pacific Islander 

Filipino 11 62.27 23.79 | 150.18 21.62 | 22.94 11.42 | 82.09 33.02 2291 3.59) 32.89 12.27 2.03 0.61 
School 

Alta Murrieta 5 52.80 3.42 62.00 19.12 | 18.37 2.65 16.80 16.10 15.80 1.92) 37.64 2.14 1.40 0.10 
Avaxat 17 62.53 22.13 | 139.06 33.22 | 25.04 9.82 73.76 39.21 24.24 3.68 31.98 9.64 1.94 0.23 
Buchanan 15 44.07 7.97 | 114.47 49.33 | 15.97 3.96 69.60 40.42 2207 3.81 33.45 2.78 1.32 0.26 
Curran 8 62.00 7.01 | 106.38 45.83 20.37 3.12 53.25 38.87 19.00 3.38 38.50 4.44 1.61 0.07 
Monte Vista 24 64.08 20.49 | 141.29 31.90 | 24.35 8.08 71.42 41.46 2296 4.09 34.64 10.46 1.86 0.21 
Murrieta 13 69.54 19.19 | 138.46 47.26 | 24.29 7.14 | 67.00 3489 21.15 2.82) 36.48 9.53 1.90 0.24 
Rail Ranch 15 70.27 20.42 | 149.27 26.67 | 24.27 7.85 | 77.53 33.88 | 23.13 1.06) 3436 965) 2.04 0.18 
Shivela 29 65.93 27.99 | 145.48 34.80 | 26.65 10.72 67.79 37.04 25.10 4.21) 33.39 12.25 1.96 0.26 
Thompson 24 67.29 33.94 | 112.62 58.19 | 18.51 5.71 | 58.58 45.75) 16.33 5.95) 3492 12.49 1.92 0.47 
McElhinney 6 75.00 32.77 | 121.33 43.50 | 32.71 15.56 | 47.17 45.27) 2433 5.47) 35.83 13.11 2.04 0.23 
Warm Springs 16 70.44 36.79 | 160.00 0.00 | 29.17 16.26 79.56 35.70 | 22.75 4.73 33.20 17.62 2.39 0.78 


*Data missing for 4 students. 
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